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DELIVERING  TRAINING  ASSESSMENTS  IN  A  SOLDIER-CENTERED  LEARNING 
ENVIRONMENT:  YEAR  ONE 

EXECUTIVE  SUMMARY 


Research  Requirement: 

Soldiers  need  to  be  trained  to  operate  in  a  complex  political  world,  with  threats  that  are 
more  ambiguous  and  unpredictable  than  ever  before.  This  context  requires  Soldiers  to  be  quick 
learners,  both  agile  and  innovative  in  their  thinking.  Further,  the  training  the  Army  provides 
must  fit  the  needs  of  Soldiers,  who  are  simultaneously  practicing  their  profession  and  expanding 
their  understanding  of  it.  Given  this  complex  situation,  the  Army  training  environment  will  be 
most  beneficial  when  it  is  adaptable  to  the  learning  needs  of  specific  learners,  with  each  working 
from  his  or  her  own  personal  and  professional  perspective.  The  Army  Learning  Model  (ALM) 
has  responded  to  these  challenges  by  presenting  a  strong  case  for  a  learner-centered  approach  to 
training,  making  use  of  viable  techniques  and  technologies  to  support  the  learning  process  (U.S. 
Army  Training  and  Doctrine  Command,  2011). 

In  support  of  the  ALM  and  given  these  challenges,  the  U.S.  Army  Research  Institute  for 
the  Behavioral  and  Social  Sciences  developed  exemplar  training  using  mobile,  virtual  classroom, 
and  virtual  collaborative  game -based  technologies  to  deliver  Soldier-centered  training  with 
integrated  assessments.  The  training  prototypes  were  developed  with  the  goal  of  supporting 
research  on  assessment  and  training  delivery  strategies  in  support  of  the  ALM.  This  report 
describes  the  prototype  training  and  assessments,  the  methodology  used  to  develop  the  training 
prototypes  and  assessments,  the  results  of  a  beta  test,  and  the  lessons  learned  from  the 
development  process. 

Procedure: 

The  development  team  used  the  Analyze,  Design,  Develop,  Implement,  and  Evaluate 
(ADDIE)  instructional  systems  design  (ISD)  approach  to  develop  the  exemplar  training  and 
assessments.  The  first  step  involved  selecting  and  analyzing  current  training  that  would  serve  as 
the  training  content  for  the  training  prototypes.  The  selected  training  concerned  the  assembly, 
configuration,  programming,  and  operation  of  a  widely  used  combat  radio,  the  AN/PRC- 148 
Joint  Enhanced  Multiband  Inter  /  Intra  Team  Radio  (JEM).  Currently,  training  on  the  JEM  is 
conducted  over  the  course  of  one  day  at  the  Signal  Regimental  Non-Commissioned  Officer 
(NCO)  Academy  at  Fort  Gordon,  Georgia.  In  transforming  the  classroom-based  JEM  training  to 
mobile,  virtual,  and  collaborative  training,  the  research  team  proposed  a  highly  integrated 
approach.  Our  general  concept  was  that  training  modules  would  build  upon  one  another  and  the 
assessments  would  be  used  to  track  learning  within  and  between  modules.  All  components  were 
informed  by  a  best  practices  review  that  was  conducted  as  part  of  this  project.  In  addition,  the 
components  were  coordinated  and  aligned  by  an  architecture  and  ISD  map.  The  training  and  the 
assessments  were  created  based  upon  the  current  classroom  training,  JEM  manuals,  and 
instructor  observation.  The  training  content  and  assessment  items  were  reviewed  by  subject 
matter  experts  (SMEs)  on  two  separate  occasions,  and  assessment  items  were  also  content 
validated.  The  end  result  was  the  creation  of  training  content  housed  in  the  respective  training 


iii 


environments  (i.e.,  mobile  device,  virtual  classroom,  and  game-based  collaborative  scenario),  as 
well  as  the  creation  and  use  of  multiple  assessment  types  (i.e.,  CAT,  checks  on  learning,  and 
individual  and  team  assessments  within  the  collaborative  scenario).  The  use  of  these  platforms 
for  JEM  training  was  then  beta-tested  by  SMEs  at  Fort  Gordon,  GA.  Upon  completion  of  each 
training  modality,  participants  completed  an  anonymous  online  survey  designed  to  measure  their 
reactions,  specifically,  satisfaction  with  technology,  satisfaction  with  instructional  design, 
satisfaction  with  learning,  utility  /  transfer  beliefs,  and  enjoyment. 

Findings: 

Results  of  the  beta-test  showed  Soldiers’  reactions  to  the  prototype  training  were 
generally  positive.  Approximately  80%  of  participants  were  satisfied  with  the  training  content 
and  rated  the  instructional  design  as  favorable  on  the  mobile  and  virtual  classroom  modalities. 
Attitudes  were  less  favorable  on  the  collaborative  scenario  modality  -  only  slightly  over  60%  of 
participants  indicated  favorable  reactions  to  collaborative  technology  across  all  items.  Further 
analyses  revealed  several  usability  issues  that  were  likely  explanations  for  the  lower  ratings. 
These  concerns  were  addressed  in  subsequent  refinements  to  the  training  prototypes.  Captured 
lessons  learned  from  the  development  effort  included  challenges  and  strategies  for  transitioning 
existing  training  content  into  a  virtual  format  and  designing  assessments  for  training. 

Utilization  of  Findings  and  Conclusions: 

The  prototypes  developed  for  this  research  program  demonstrate  how  maturing 
technologies  including  mobile  and  virtual  learning  environments  can  be  used  to  deliver 
individual,  classroom,  and  collaborative  training  in  line  with  the  AFM.  The  prototypes  also 
demonstrate  how  assessments  can  be  integrated  within  these  technologies  to  provide  targeted 
feedback  and  track  learning  across  and  between  platforms.  The  prototype  training  will  be  used 
in  future  research  to  examine  critical  issues  regarding  the  use  of  assessments  in  a  Soldier- 
centered  learning  environment.  The  prototype  technologies  from  this  effort  have  been  briefed  to 
representatives  at  the  U.S.  Training  and  Doctrine  Command  and  shared  at  the  Annual  Meeting  of 
Society  of  Industrial  and  Organizational  Psychologists  and  the  Interservice  /  Industry  Training, 
Simulation  and  Education  Conference. 
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DELIVERING  TRAINING  ASSESSMENTS  IN  A  SOLDIER-CENTERED 
LEARNING  ENVIRONMENT:  YEAR  ONE 

Introduction 

Today’s  Soldiers  face  multiple  deployments,  forward  stationing  worldwide,  and  a 
dynamic  geopolitical  environment  that  poses  numerous  threats  that  are  more  ambiguous  than 
ever  before.  The  complexities  and  challenges  of  the  modern  military  environment  have 
significant  implications  for  the  way  Soldiers  operate,  and  in  turn,  the  way  they  are  trained.  For 
example,  a  dynamic  threat  environment  means  that  there  are  ever-changing  knowledge  and  skill 
sets  that  Soldiers  must  master  and  do  so  quickly.  Varied  and  rapidly  changing  operational 
environments  require  lessons  learned  in  the  field  to  be  adapted  quickly  in  training  and  education. 
The  implications  of  these  challenges  are  that  the  time  spent  on  training  and  educating  Soldiers  as 
well  as  assessing  them  must  be  optimally  efficient;  training  and  educational  opportunities  must 
be  flexible  enough  to  accommodate  irregular  schedules  and  heterogeneous  environments;  and 
content  and  delivery  methods  must  be  accessible,  intuitive,  instructionally  sound,  and  incorporate 
lessons  learned  in  as  near  to  real-time  as  possible. 

As  part  of  the  response  to  these  challenges,  the  U.S.  Army  Learning  Model  (ALM)  sets 
forward  an  agenda  for  innovation  in  Army  training,  where  instructor-centered  training  is 
replaced  by  learner-centered  training  (U.S.  Army  Training  and  Doctrine  Command,  201 1).  The 
ALM  focuses  on  the  importance  of  anytime,  anywhere  training  that  has  the  ability  to  actively 
engage  trainees,  from  recmits  to  retirees,  with  a  learner-centered  approach.  Rather  than  limiting 
training  to  specific  timeframes  and  locations  (e.g.,  a  ‘brick  and  mortar’  training  environment), 
the  ALM  calls  for  a  training  system  that  can  be  accessed  at  the  ‘point  of  need’  and  one,  “. .  .that 
extends  knowledge  to  Soldiers  at  the  operational  edge,  is  capable  of  updating  learning  content 
rapidly,  and  is  responsive  to  Operational  Army  needs”  (U.S.  Army  Training  and  Doctrine 
Command,  2011,  p.  16). 

A  key  element  of  this  vision  is  the  use  of  distributed  learning  technology  to  support 
learning  anywhere  and  anytime.  Distributed  learning  can  include  synchronous  training,  such  as 
training  in  a  virtual  classroom,  and  asynchronous  training,  such  as  self-directed  learning  on  a 
mobile  device.  Research  on  mobile  learning  cites  a  number  of  potential  advantages  for  mobile 
learning  technology,  including  more  flexible  access  to  training  material  which  can  increase 
learning  gains  (e.g.  Holden  &  Sykes,  2011;  Norris  &  Soloway,  2004;  Roschelle  &  Pea,  2002; 
Soloway,  Norris,  Blumenfeld,  Fishman,  Krajcik,  &  Marx,  2001).  Further,  by  shifting  training 
devices  into  the  hands  of  Soldiers  outside  the  classroom,  it  contextualizes  learning  in  the  real 
world  which  situated  cognition  theory  predicts  should  be  beneficial  (Lave,  1988). 

Training  and  education  literature  also  discusses  the  benefits  of  leveraging  videogames 
and  virtual  worlds  for  training  environments.  Specifically,  research  shows  engagement  in  either 
a  videogame -based  training  environment  or  a  virtual  world  (such  as  the  multi-user  virtual 
environment  [MUVE]  Second  Life )  can  increase  trainee  motivation  and  engagement,  and  thus 
increase  the  desire  to  learn,  the  desire  to  practice,  and  training  relevant  outcomes  (i.e., 
knowledge  or  procedural  skill)  (Chang,  Giitl,  Kopeinik,  &  Williams,  2009;  Mautone,  Spiker, 
Karp,  &  Conkey,  2010;  Topolski,  Leibrecht,  Cooley,  Rossi,  Lampton,  &  Knerr,  2010).  Further, 
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as  a  result  of  their  coding  architecture,  both  videogames  and  virtual  worlds  have  the  ability  to 
supply  learners  with  timely  feedback  with  customizable  levels  of  specificity  (Mautone  et  al., 
2010).  This  does  not  imply  that  feedback  features  are  native  to  videogames  and  virtual  worlds. 
Developing  a  coding  architecture  that  would  allow  timely  feedback  with  customizable  levels  of 
specificity  necessitates  a  time  investment  and  requires  expertise  in  learning  theory,  and  computer 
science.  However,  with  an  appropriate  coding  architecture  in  place,  videogames  and  virtual 
worlds  can  provide  trainees  with  or  immediate,  customizable  feedback  based  on  real-time 
assessment  of  trainee  performance. 

In  addition  to  these  training  platforms,  the  ALM  also  mentions  the  importance  of  using 
valid  and  reliable  assessments  to  ensure  learning  has  occurred  to  a  standard,  and  as  a  way  to 
tailor  training.  The  ALM  specifically  mentions  pre-/post-tests  as  one  way  to  gauge  learning,  but 
assessments  can  be  incorporated  throughout  the  training  as  well.  A  variety  of  assessment  types 
beyond  traditional  multiple-choice  tests  may  be  used  to  assess  learning.  For  example,  computer- 
adaptive  tests  (CATs)  can  be  used  to  reduce  test  length  without  losing  test  precision 
(Triantafillou,  Georgiadou,  &  Economides,  2008).  Situational  judgment  tests  can  be  an  effective 
way  to  measure  more  complex  application  of  skill  or  knowledge,  and  360-degree  assessments 
can  be  helpful  in  measuring  softer,  more  interpersonal  or  collaborative  skills.  Simulated  tasks 
(or  “work  samples”)  offer  a  high  fidelity  option  for  measuring  performance  that  can  occur  at  the 
individual  or  team  level.  The  ALM  focuses  on  assessing  individual  learning,  but  given  the 
Army’s  emphasis  on  collective  training  and  performance,  assessment  of  unit-based  skills  and 
performance  must  be  considered  as  well. 

The  current  project  focused  on  creating  and  testing  training  and  assessment  prototypes  in 
support  of  the  ALM.  Specifically,  the  objective  was  to  create  prototype  technology-based 
training  with  embedded  assessments  for  subsequent  research.  This  included  an  investigation  of 
approaches  to  training  and  assessment  development  and  delivery  using  mobile,  virtual 
classroom,  and  virtual  collaborative  technologies.  Although  the  ALM  specifies  the  need  for 
learner-centered,  ubiquitous  training,  the  model  itself  does  not  specify  how  training  and 
assessments  should  be  developed,  designed,  delivered,  and  otherwise  used  to  maximize  Soldier 
training.  Thus,  questions  surrounding  the  best  way  to  deliver  assessments,  the  value  of  using 
adaptive  assessments,  and  the  optimal  assessment  frequency  remain  unanswered.  This  paper 
discusses  the  development  of  the  prototype  training  and  assessments,  including  a  discussion  of 
the  prototype  concept,  the  instructional  design  approach  used  to  develop  the  training  and 
corresponding  assessments,  and  the  technology  considerations  and  constraints.  The  paper  also 
describes  the  results  of  a  beta  test  that  examined  the  validity  and  usability  of  the  training 
platforms  and  assessments.  It  concludes  with  a  discussion  of  future  research,  which  examines 
critical  questions  regarding  the  design  and  delivery  of  assessments  within  the  prototype  training. 

Throughout  the  report,  we  highlight  features  or  methods  that  were  leveraged  from  a  best 
practices  review  conducted  as  part  of  this  effort  (see  Brusso,  Wisher,  Paddock  &  Hatfield,  2014). 
This  review,  which  focused  on  the  use  of  assessments  in  cross-platform,  emergent  technology 
training  environments,  involved  a  substantive  literature  review  and  numerous  interviews  with 
subject  matter  experts  (SMEs)  who  were  well  versed  in  developing  and  integrating  training  and 
assessments  into  technology-based  training  platforms.  The  results  yielded  training  exemplars 
and  best  practices  that  showcased  how  training  and  assessments  could  be  integrated  across 
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training  technologies.  Examples  of  these  best  practices  included  applying  principles  of  learning 
sciences,  creating  and  using  an  assessment  map  to  align  assessments  to  learning  objectives, 
employing  frequent  testing  to  reinforce  learning,  and  embedding  assessments  into  the  training 
platform. 

The  following  sections  describe  the  prototype  technologies  and  provide  an  overview  of 
the  prototype  concept. 


Prototype  Descriptions 

The  prototype  training  technologies  developed  for  this  effort  were  designed  to  teach 
Soldiers  how  to  assemble,  configure,  program  and  troubleshoot  a  widely  used  combat  radio,  the 
AN  /  PRC- 148  Joint  Enhanced  Multiband  Inter  /  Intra  Team  (JEM)  radio.  The  widespread  use  of 
the  JEM  radio  in  the  Army,  as  well  as  other  services,  made  it  an  appropriate  candidate  for  the 
prototype  training.  Currently,  training  for  the  JEM  radio  is  conducted  in  one  day  at  the  Signal 
Regimental  Non-Commissioned  Officer  (NCO)  Academy  at  Fort  Gordon,  Georgia.  During  the 
one-day  course,  a  class  of  approximately  30  NCOs  participates  in  PowerPoint-based  didactic 
instruction,  followed  by  hands-on  practice  with  the  radio.  The  duration  of  the  instruction  and 
hands-on  practice  sections  typically  total  six  hours  (three  hours  each),  but  can  vary  as  a  result  of 
class  size  and  instructor  lecture  speed.  The  following  day,  students  take  an  open-book  test  to 
determine  whether  or  not  they  pass  or  fail  that  section  of  the  course. 

In  transforming  the  current  JEM  training  from  classroom-based  instruction  to  mobile, 
virtual,  and  collaborative  training  (and  the  corresponding  suite  of  assessments),  the  team  used  an 
integrated  approach.  Our  concept  for  the  training  and  assessments  was  for  the  training  presented 
in  each  platform  (i.e.,  mobile,  virtual  classroom,  and  virtual  collaborative  scenario)  to  build  upon 
one  another  and  for  the  assessments  to  track  learning  within  and  between  modules.  Specifically, 
a  CAT  was  to  be  used  to  track  learning  progress  between  the  mobile  and  the  virtual  classroom 
training.  Interim  assessments,  referred  to  as  check  on  learning  activities  were  used  to  measure 
learning  within  each  platform.  A  collaborative,  capstone-like  exercise  was  used  to  measure 
learning  and  performance  at  the  end  of  training  at  both  the  individual  and  team  levels.  All 
components  were  coordinated  and  aligned  using  an  overarching  ISD  map  and  a  comprehensive 
technology  architecture  plan.  All  portions  of  the  training  were  integrated  into  the  Soldier- 
Centered  Army  Learning  Environment  (SCALE),  a  prototype  data-driven  training  architecture 
and  learning  management  system  (Mangold,  Beauchat,  Long,  &  Ambum,  2012).  Figure  1 
provides  an  overview  of  the  prototype  concept. 
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Figure  1:  Protoype  concept. 


Mobile  Training  Prototype 

The  mobile  application  was  designed  to  teach  introductory  facts  and  concepts  regarding 
the  radio  through  a  combination  of  didactic  learning  content  and  simple  hands-on  use  of  a  virtual 
radio.  Although  the  training  was  designed  to  run  on  iOS  devices,  specifically  the  iPad,  the  core 
content  and  technologies  are  portable  to  other  tablet  devices,  web-based  and  stand-alone  PC  and 
Mac  environments.  The  didactic  learning  content  includes  narrated  videos,  text,  images,  and 
animation  that  review  key  features  and  functionality  of  the  JEM  radio.  The  mobile  training  also 
contains  “try-it”  activities  that  leverage  a  virtual  JEM  radio  (see  Figure  2).  Students  can 
manipulate  the  buttons  and  knobs  on  the  virtual  radio  using  similar  gestures  as  they  would  when 
operating  a  real  radio.  A  key  feature  of  the  mobile  training  is  the  integration  of  assessments  (i.e., 
checks  on  learning)  throughout  the  training  content.  These  assessments  are  presented  in  formats 
ranging  from  traditional  multiple-choice,  true  /  false,  and  matching  questions,  to  exercises  using 
a  virtual  3D  JEM  radio.  The  purpose  of  the  assessments  is  to  reinforce  the  material  covered  in 
training  and  provide  students  with  feedback  regarding  their  current  level  of  understanding. 
Students  can  use  this  feedback  to  correct  any  misconceptions  they  may  have  regarding  the  use 
and  operation  of  the  radio.  For  example,  students  receive  corrective  feedback  when  they 
incorrectly  respond  to  a  multiple  choice,  true  /  false  or  matching  question.  After  receiving  this 
information,  students  proceed  to  the  next  question  (they  are  not  provided  an  additional 
opportunity  to  answer  the  question).  For  items  that  target  procedural  knowledge,  such  as 
exercises  that  use  the  virtual  JEM  radio,  students  receive  step-based  corrective  feedback. 
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However,  unlike  the  multiple-choice  questions,  students  are  required  to  demonstrate  mastery  of 
the  procedure  before  continuing  to  the  next  question. 

As  shown  in  Figure  1,  pre-and  post-tests  (using  a  CAT)  bookend  the  mobile  training. 

The  purpose  of  the  CAT  is  to  measure  knowledge  gains  as  a  result  of  training.  After  completing 
the  mobile  training,  students  progress  to  the  virtual  classroom  training.  Progression  to  the  virtual 
classroom  is  not  contingent  on  performance  on  the  CAT  (although  the  CAT  could  be  used  for 
this  purpose).  Assessment  scores  from  the  interim  knowledge  checks  and  the  CAT  are  recorded 
and  stored  in  the  SCALE  architecture,  as  a  means  for  tracking  learning. 
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The  On  /  Off  Switch,  which  is  located  at  the  top  left  of  the  radio  closest  to  the  front, 
serves  several  purposes.  First,  by  turning  It  to  the  right,  you  power  on  the  radio.  You 
will  only  power  on  the  radio  when  it  is  completely  assembled. 

The  On  /  Off  Switch  also  serves  as  the  Volume  Control.  The  rotary  switch  contains 
distinct  notches  to  control  the  volume.  In  a  situation  when  the  output  must  be  low, 
you  do  not  want  the  ' 


Figure  2:  Mobile  platform  screen-shot  showing  animated  sequence  to  turn  the  radio  on  and  off. 

Virtual  Classroom 

The  virtual  classroom  is  designed  to  extend  core  aspects  of  the  mobile  training  to  include 
the  ability  to  share  and  deliver  content  through  the  use  of  a  “virtual  schoolhouse”  style  interface 
(see  Figure  3).  The  training  material  covered  in  this  phase  of  training  addresses  more  complex 
skills  such  as  advanced  programming  and  troubleshooting  procedures.  The  virtual  classroom 
application  allows  the  instructor  to  select  and  interact  with  training  content  that  is  delivered 
synchronously  to  students’  workstation.  Content  includes  text  and  imagery,  as  well  as  3D 


5 


content  that  can  be  manipulated  by  the  instructor,  which  enables  the  instructor  to  demonstrate 
how  to  manipulate  the  radio.  While  in  the  classroom,  learners  are  able  to  communicate  (through 
text-based  chat  and  voice  communication)  with  their  instructors  and  peers,  and  request  instructor 
assistance  via  a  “help”  button. 

Throughout  the  virtual  classroom  the  instructor  is  able  to  present  the  learners  with  a  set  of 
interactive  checks  on  learning  activities  that  include  multiple  choice  questions,  image  matching, 
and  interactive  radio  programming  questions.  Students  also  receive  corrective  feedback  during 
these  activities.  Upon  completion  of  the  virtual  classroom  instruction  (and  corresponding 
assessments),  students  complete  another  CAT.  Similar  to  the  mobile  training  experience,  student 
performance  on  the  check  on  learning  activities  and  CAT  is  collected  and  stored  in  the  SCALE 
architecture.  After  completing  the  CAT,  students  proceed  to  the  collaborative,  scenario-based 
training  exercise. 


Figure  3:  Virtual  classroom  platform  (Instructor  View)  screen-shot. 

Collaborative  Scenario 

The  collaborative  application  extends  features  of  the  mobile  application  and  virtual 
classroom  to  include  realistic  scenario-driven  vignettes  that  require  students  work  in  teams  to 
collaboratively  resolve  issues  related  to  manipulating  and  troubleshooting  the  JEM  radio  (see 
Figure  4).  During  the  exercise,  the  control  of  the  radio  rotates  among  team  members  (teams 
consist  of  three  players)  so  that  everyone  has  a  chance  to  manipulate  the  radio.  Like  most 
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collaborative  game-based  exercises,  all  players  see  the  same  screen.  However,  only  one  team 
member  is  able  to  “use’  the  radio  while  the  other  two  players  watch  and  offer  input.  The  team 
member  in  charge  of  the  radio  for  the  given  vignette  is  designated  as  the  “active  learner.” 
During  the  vignettes,  players  have  the  ability  to  communicate  with  each  other  using  Voice  Over 
Internet  Protocol  (VoIP).  Team  members  can  use  this  communication  feature  for  discussion  as 
they  proceed  through  the  vignettes  and  to  collaborate  about  how  to  complete  a  task  using  the 
radio. 


Figure  4:  Collaborative  scenario  screen-shot. 


The  collaborative  scenario  includes  both  group  and  individual  level  assessments.  Groups 
are  assessed  as  a  whole  during  the  vignettes  where  the  overall  score  is  determined  by  the  number 
of  successful  radio  actions  completed  by  each  team  member.  Thus  team  performance  is  an 
aggregate  score  of  group  performance.  In  addition,  players  complete  individual  knowledge- 
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based  assessments  as  they  progress  through  the  collaborative  scenario.  These  items  are  scored  at 
the  individual  level.  Individual  and  team  level  scores  are  collected  and  stored  in  the  SCALE 
architecture. 


Prototype  Design  and  Development 

The  development  of  the  prototype  training  and  corresponding  assessments  occurred 
concurrently  and  with  ongoing  coordination  between  the  ISD  team  and  the  assessment  team.  All 
material  produced  by  one  team  was  iteratively  reviewed  by  the  other  team,  and  vice  versa. 
Prototype  development  is  described  in  the  following  sections: 

•  Overall  training  content  design, 

•  Training  environment  development,  and 

•  Assessment  development. 

During  the  course  of  this  project,  the  research  team  visited  the  Signal  NCO  Academy  at 
Fort  Gordon,  GA  multiple  times  to  interview  instructors,  review  course  materials,  observe 
classes,  and  collect  data  to  help  develop  the  JEM  ISD  Map,  prototype  trainings,  and  assessments. 
Because  all  three  training  modalities  (i.e.,  mobile,  virtual,  and  collaborative)  and  the 
corresponding  assessments  were  created  concurrently,  these  data  collection  trips  typically  served 
multiple  purposes.  Table  1  briefly  summarizes  these  visits.  Throughout  this  report,  we  refer 
back  to  this  exhibit  as  additional  detail  is  provided  about  the  data  collections  pertaining  to  each 
prototype. 

Table  1 


Summary  of  Data  Collection  Trips 


Trip 

Date 

SMEs  /  Participants  at  the  Signal  NCO 
Academy  at  Fort  Gordon 

Purpose  /  Activities 

1 

25-26  January  2012 

2  instructors 

Preliminary  collection  of 
information 

2 

13-14  March  2012 

1  class  with  2  instructors  and 
approximately  30  learners 

Observation  of  JEM 
instructor-led  training 

3 

24-25  April  2012 

3  instructors 

SME  review  workshops 

4 

6-9  August  2012 

3  instructors  and  8  students  (MOS 

25U) 

Content  validation 
workshops 

5 

21-24  January  2013 

21  learners  (MOS  25C  and  25U) 

Beta  test 

*Note.  MOS  25C  refers  to  Military  Occupational  Specialty  Radio  Operator  /  Maintainer,  25U  refers  to  Signal 
Support  Systems  Specialist. 


Overall  Training  Content  Design 

The  first  step  in  this  project  was  to  determine  how  to  best  represent  the  content  covered  in 
the  classroom-based  JEM  training.  To  do  this,  a  systematic,  learning-objective  driven  ISD 
process  was  used,  based  on  the  ADDIE  Model  (Analysis,  Design,  Development,  Implementation, 
and  Evaluation)  and  fully  supported  by  the  Systematic  Instructional  Design  model  (Dick  &  Cary, 
1990).  The  process  was  consistent  with  the  best  practices  findings  and  provisional  guideline  to 
“apply  principles  of  the  learning  sciences  to  carefully  plan  the  integration  of  learning  experiences 
across  training  platforms  to  ensure  the  effectiveness  and  efficiency  of  the  training,”  (Brusso  et 
al„  2014,  p.  20). 

At  the  start  of  the  project,  the  ISD  team  identified  the  learning  objectives  related  to  the 
assembly,  configuration,  programming,  and  operation  of  the  JEM.  The  goal  was  to  identify 
learning  needs  for  the  JEM  and  then  decompose  behavioral  and  cognitive-based  learning 
objectives  in  a  systematic  manner  using  terminal  ( macro-level ),  sub-ordinate  ( micro-level ),  and 
enabling  learning  objectives  ( nano-level )  to  develop  a  detailed  ISD  map  (see  Figure  5).  The 
decomposition  of  the  systematic  learning  objectives  (terminal,  sub-ordinate,  and  enabling) 
produced  an  instructionally  sound  design. 
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Figure  5:  ISD  Map  schema. 


To  identify  learning  objectives  and  develop  this  map,  use  of  the  radio  was  observed 
during  data  collection  trip  1  (see  Table  1:  Summary  of  Data  Collection  Trips).  The  following 
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background  materials  provided  by  the  course  instructor  and  SMEs  at  Fort  Gordon  were  also 
carefully  reviewed: 

•  Thales  Manufacturer’s  Technical  Manual  for  the  AN/PRC- 148  (JEM). 

•  The  AN/PRC- 148  Instructor  Slides  used  at  Fort  Gordon  for  the  Signal  Regimental  NCO 
Academy’s  Advanced  Leader  Course  (ALC). 

After  gathering  and  analyzing  this  information,  the  ISD  team  developed  the  first  draft  of 
the  JEM  ISD  Map.  This  map  was  updated  several  times  over  the  course  of  the  project,  including 
after  observing  the  classroom-based  training  (Trip  2)  and  after  meeting  with  SMEs  and  Small 
Group  Leaders  (SGLs)  to  review  the  training  storyboards  and  draft  assessment  items  (Trips  3  and 
4). 


The  final  ISD  Map  contains  the  entire  learning  objective  decomposition  along  with 
associated  training  modalities  (Appendix  A).  An  inspection  of  the  map  reveals  a  natural 
progression  of  skills  starting  with  basic  content  familiarization  and  evolving  to  higher-order 
cognitive  skills  such  as  critical  thinking  and  peer  evaluation.  The  ISD  map  also  revealed  a 
categorization  of  these  skills  at  the  terminal  objective  level  (see  Figure  6). 


Main  Instructional  Goal  1.0 

Learner  will  demonstrate  the  proper  use  of  the  JEM  with  100%  accuracy  and  within 
rapid  timeframes  in  both  usual  and  unusual  conditions  and  under  duress  and 
constraints,  while  troubleshooting  issues  as  they  arise. 


Terminal  Objective  1.1 

Learner  will  describe  the  basic  features,  technical 
components  and  capabilities  of  the  JEM,  identify 
and  explain  the  purpose  of  each  component, 
identify  and  explain  how  to  connect  external 
components,  and  recognize  problems  that  require 
general  troubleshooting. 


Mobile  Device 


Terminal  Objective  1.2 

Learner  will  demonstrate  accurate 
and  timely  operation  of  the  JEM  in 
usual  conditions,  including 
troubleshooting  as  required  in 
specific  circumstances. 


Terminal  Objective  1.3 

Learner  will  demonstrate  operation  of  the  JEM, 
including  troubleshooting  as  required,  and 
evaluate  other  team  members'  use  of  the  JEM 
through  guidance,  collaboration  and  after  action 
review,  in  usual  and  unusual  conditions. 


Virtual 

Classroom 


Collaborative 

Scenario 


Figure  6:  ISD  Map  for  terminal  objectives. 


The  three  Terminal  Objectives  optimally  aligned  with  the  three  different  training  delivery 
methods  (i.e.,  mobile  device,  virtual  classroom,  and  collaborative  scenario  exercise)  due  to  the 
affordances  of  these  training  modalities  and  the  characteristics  necessitated  by  the  objectives. 
Bower  (2008)  discussed  the  importance  of  understanding  the  match  between  learning  objectives 
and  affordances  of  training  delivery  methods.  For  example,  temporal  affordances  (i.e.,  the 
ability  to  access  content  anytime  and  anywhere),  media  affordances  (i.e.,  read-ability,  view- 
ability,  listen-ability,  watch-ability),  and  spatial  affordances  (i.e.,  move-ability)  were  a  few 
features  that  aligned  with  Terminal  Objective  1.1.  Alternatively,  demonstration  was  not  relevant 
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to  Terminal  Objective  1.1.  Thus,  the  enabling  objectives  under  Terminal  Objective  1.1  optimally 
aligned  with  a  Mobile  Application,  as  this  delivery  method  provided  the  requisite  capabilities. 
Enabling  Objectives  under  Terminal  Objective  1.2,  conversely,  necessitated  demonstration. 
Therefore,  Terminal  Objective  1.2  more  appropriately  aligned  with  a  Virtual  Classroom  delivery 
method  which  allowed  an  instructor  to  demonstrate  the  requisite  skills  in  an  in-depth  manner. 

As  shown  in  the  ISD  Map,  each  learning  objective  was  associated  with  an  identifier 
number  that  allowed  the  design  team  to  align  all  aspects  of  the  prototypes  to  learning  objectives 
including  didactic  content,  scaffolding  content  (such  as  hints  or  guidance),  customized  real-time 
performance  feedback,  and  summative  feedback.  The  ISD  map  also  allowed  the  design  team  to 
easily  map  the  assessments  to  the  training  components,  ensuring  that  the  assessments  accurately 
reflected  the  training  content.  This  use  of  an  “assessment  map”  is  consistent  with  the  findings 
from  the  best  practice  report  (Brusso  et  al.,  2014). 

Training  Environment  Development 

The  ISD  Map  allowed  the  design  team  to  work  from  a  common  schema  while  designing 
and  developing  the  training  and  assessment  prototypes.  It  served  as  a  common  language  for  all 
team  members  when  referring  to  learning  objectives  at  any  level.  In  developing  the  prototypes, 
two  questions  were  particularly  important  to  consider:  How  do  you  optimize  platform  design  for 
each  particular  training  platform?  And,  how  do  embed  assessments  in  the  respective  platforms? 
The  following  sections  address  how  we  answered  these  questions  by  describing  the  development 
of  the  training  platforms  and  embedded  assessments. 

Mobile  environment.  The  mobile  application  developed  for  this  effort  was  designed  to 
meet  the  learning  objectives  under  Terminal  Objective  1.1.  Specifically  it  was  designed  to  teach 
students  how  to:  identify  and  describe  the  basic  features,  components;  and  capabilities  of  the 
JEM  radio;  identify  and  explain  how  to  connect  external  components  to  the  radio;  and  recognize 
problems  that  require  general  troubleshooting. 

Mobile  training  content.  Content  for  the  mobile  application  was  developed  using  the 
following  sources: 

•  JEM  Technical  Manual, 

•  JEM  classroom  presentation  slides  used  for  the  ALC  training  at  Fort  Gordon,  and 

•  Interview  and  observation  data  collected  during  data  collection  Trips  3  and  4. 

The  subordinate  objectives  presented  in  Table  2  were  identified  for  the  mobile 
application.  These  objectives  translated  into  four  modules. 
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Table  2 


Mobile  Application  Subordinate  Objectives 


Subordinate  Objective 

Description 

1.1.1 

Learner  will  define  relevant  terminology,  explain  radio  safety 
considerations,  and  describe  features,  capabilities,  and 
specifications  of  the  JEM. 

1.1.2 

Learner  will  identify  the  JEM’s  components  (including  external 
components),  explain  how  they  are  properly  assembled  and 
maintained,  describe  how  they  are  used  in  the  context  of  the 
radio’s  functionality,  and  state  any  relevant  information  such  as 
reminders  or  troubleshooting. 

1.1.3 

Learner  will  explain  and  demonstrate  at  a  high  level  how  to 
configure  and  operate  the  JEM  (including  KeyFill)  as  well  as 
how  to  access  each  display  screen  and  explain  the  significance 
of  each. 

1.1.4 

Learner  will  list  cues  that  indicate  an  issue  with  the  radio  and 
explain  how  to  troubleshoot  general  issues  that  can  commonly 
arise. 

The  ISD  Map  displays  how  these  four  subordinate  objectives  were  decomposed  into 
specific  enabling  objectives  such  as  "Describe  the  use  of  the  lamp  function  and  explain  when  and 
when  not  to  use  it."  These  enabling  objectives  were  then  aligned  with  specific  didactic  content, 
as  well  as  check-on-learning  and  CAT  items  (discussed  in  the  Assessments  section  to  follow). 

As  previously  noted,  content  was  leveraged  from  the  current  training  slides  and  the 
technical  manual.  As  content  was  compiled,  it  was  reviewed  for  accuracy  by  SMEs  during  the 
third  and  fourth  data  collections.  Revisions  were  made  after  each  of  these  data  collections  based 
on  SME  feedback. 

Mobile  technical  platform.  The  next  step  in  development  of  the  application  was  to 
determine  which  technologies  would  be  used  in  creating  the  mobile  application.  Table  3 
provides  detailed  descriptions  of  each  technology  used. 
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Table  3 


Mobile  Application  Technology  Descriptions 


Technology 

Description 

Unity3D 

A  3D  game  and  application  development  environment  that 
leverages  the  C#  (“C-Sharp”)  language  and  current  open  2D  / 

3D  art  asset  standards.  Used  to  develop  all  classroom  server  and 
client  applications.  Unity  was  identified  as  the  platform  of 
choice  due  to  its  ability  to  deploy  applications  to  multiple 
platforms  using  the  same  code  and  art  assets. 

Drasgow  Assessment  Engine 

Leveraged  the  .NET-developed  adaptive  assessment  engine  from 
Drasgow  within  Unity3D  to  present  and  manage  the  pre  and  post 
adaptive  assessments. 

*Note.  The  hardware  intended  for  the  mobile  application  is  the  iPad  first  generation  (or  greater)  with  iOS  5.1,  touch 
interface. 


Next,  storyboards  were  created  to  provide  the  technical  team  the  content  and  design  of 
the  mobile  application.  Figure  7  is  a  sample  storyboard  representing  enabling  objective  1. 1.1.3 
(Safety  Considerations  for  the  JEM).  Storyboards  were  used  to  elucidate  the  necessary  elements 
(e.g.,  objective  number,  media  required,  etc.)  and  then  integrate  elements  into  each  page  of 
content  for  the  mobile  device. 


Figure  7:  Mobile  application  storyboard. 
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Finally,  a  key  component  of  the  mobile  training  was  a  fully  functional  virtual  JEM  radio 
(see  Figure  8).  The  virtual  radio  was  developed  in  parallel  across  all  modalities  (i.e.,  the  mobile, 
virtual  classroom,  and  collaborative  scenario).  As  the  functionality  requirements  surfaced  per 
modality,  new  functionality  was  built  into  virtual  simulated  radio.  The  final  virtual  radio  was 
used  in  each  training  modality.  This  approach  provided  for  efficient  development  of  assets,  as 
well  as  ensured  that  the  virtual  radio  functioned  consistently  across  each  modality. 


Figure  8:  Screen  captures  of  virtual  radio.  Full  front  view  is  top  left, 
keypad  and  screen  is  top  right,  side  view  is  bottom  right,  and  back  view 
/  battery  connect  is  bottom  left. 
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Virtual  classroom.  The  virtual  classroom  was  designed  to  allow  the  learner  the 
opportunity  to  actively  demonstrate  how  to  assemble  /  disassemble,  configure,  program,  operate, 
and  troubleshoot  the  JEM  radio. 

Virtual  classroom  training  content.  The  ISD  Map  contains  a  detailed  breakdown  of 
virtual  classroom  objectives  (Terminal  Objective  1.2).  Content  for  the  learning  objectives  in 
Terminal  Objective  1.2  was  developed  using  the  following  sources: 

•  JEM  Technical  Manual, 

•  JEM  classroom  presentation  slides  used  for  the  ALC  training  at  Fort  Gordon,  and 

•  SME  interview  data  collected  during  data  collection  Trips  2,  3,  and  4. 

Table  4  contains  a  listing  of  the  subordinate  objectives  contained  in  the  virtual  classroom 
platform. 

Table  4 


Virtual  Classroom  Subordinate  Objectives 


Subordinate  Objective 

Description 

1.2.1 

Learner  will  demonstrate  the  JEM’s  Start  Up  procedures,  while 
trouble-shooting  as  required,  and  will  engage  in  peer-to-peer 
instruction  as  appropriate. 

1.2.2 

Learner  will  demonstrate  how  to  program  the  JEM’s  Global 
Parameters,  while  trouble-shooting  as  required,  and  will  engage 
in  peer-to-peer  instruction  as  appropriate. 

1.2.3 

Learner  will  demonstrate  the  JEM’s  Key  Management 
procedures,  while  trouble-shooting  as  required,  and  will  engage 
in  peer-to-peer  instruction  as  appropriate. 

1.2.4 

Learner  will  demonstrate  how  to  program  channels  on  the  JEM, 
while  trouble-shooting  as  required,  and  will  engage  in  peer-to- 
peer  instruction  as  appropriate. 

1.2.5 

Learner  will  demonstrate  how  to  set  up  Group  Programming, 
while  trouble-shooting  as  required,  and  will  engage  in  peer-to- 
peer  instruction  as  appropriate. 

*1.2.7 

Learner  will  demonstrate  how  to  zeroize  the  JEM  and  provide  a 
context  for  a  complete  zeroize  versus  a  channel  zeroize  and  will 
engage  in  peer-to-peer  interaction  as  appropriate. 

1.2.8 

Learner  will  demonstrate  how  to  clone  JEM’s  programming, 
while  trouble-shooting  as  required,  and  will  engage  in  peer-to- 
peer  interaction  as  appropriate. 

*1.2.10 

Learner  will  demonstrate  establishing  communications  with 
configured  radio  using  the  PTT  with  another  peer,  several 
peers,  and  /  or  with  the  virtual  instructor. 

*Note.  1.2.6,  1.2.9  remain  in  the  ISD  map,  but  were  not  included  in  the  training  content  as  they  were 
determined  to  be  out  of  scope  by  ARI  and  the  SMEs  after  reviewing  the  results  of  the  beta-test. 
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The  JEM  ISD  Map  displays  how  the  eight  subordinate  objectives  were  decomposed  into 
specific  enabling  objectives  such  as  “Identify  and  inventory  the  standard  JEM  accessories.” 
Objectives  were  then  aligned  with  specific  didactic  content.  These  objectives  were  also  aligned 
with  checks  on  learning  and  CAT  items  (discussed  in  the  Assessments  section  to  follow). 

As  with  the  mobile  training,  content  was  leveraged  from  the  current  training  slides  and 
the  technical  manual.  SMEs  reviewed  the  accuracy  the  content  during  the  second,  third,  and 
forth  data  collection  trips;  revisions  were  made  after  each  trip. 

Virtual  classroom  technical  platform.  The  virtual  classroom  employed  the  following 
software  technologies:  Unity  Master  Server,  Unity  Multiplayer  Networking  Application 
Programming  Interface  (API),  TeamSpeak  VIOP  Software  Development  Kit  (SDK),  and 
Unity3D.  The  descriptions  of  each  software  technology  are  provided  below  in  Table  5. 

Table  5 


Virtual  Classroom  Software  Technology  Descriptions 


Technology 

Description 

Unity  Master  Server 

Provided  a  classroom  coordination  service.  When  a 
classroom  server  is  started,  the  service  registers  with 
the  master  server  with  details  about  the  location  and 
type  of  server  that  has  been  made  available.  When  the 
instructor  client  or  learner  client  is  started,  it  reaches 
out  to  the  master  server  to  obtain  a  list  of  relevant 
classroom  servers  available. 

Unity  Multiplayer  Networking  API 

Provided  a  technology  layer  that  manages  network 
messaging  and  synchronization  services.  Chosen  for  its 
built-in  networking  application  programming  interface 
(API)  because  of  the  tight  integration  with  the  Unity 
development  environment. 

TeamSpeak  VIOP  SDK 

Provided  integrated  voice  communications.  The  voice 
communications  system  includes  a  standalone  voice 
server  application,  and  a  software  integration  layer  built 
into  the  virtual  classroom  clients. 

Unity3D 

A  3D  game  and  application  development  environment 
that  leverages  the  C#  (C-Sharp)  language  and  current 
open  2D  /  3D  art  asset  standards.  Used  to  develop  all 
classroom  server  and  client  applications.  Unity  was 
identified  as  the  platform  of  choice  due  to  its  ability  to 
deploy  applications  to  multiple  platforms  using  the 
same  code  and  art  assets. 

As  mentioned  previously,  the  virtual  radio  was  developed  in  parallel  across  all 
modalities;  new  functionality  was  built  into  it  based  on  requirements  of  the  virtual  classroom.  A 
script  was  created  to  provide  the  technical  team  the  content  and  design  of  the  virtual  classroom. 
The  script  included  details  about  learner  interaction  and  screen  activity  sequences.  This 
approach  was  similar  to  the  storyboard  approach  used  for  the  mobile  platform;  however,  the 
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mobile  storyboards  only  contained  learning  and  media  content,  whereas  the  virtual  classroom 
scripts  contained  screen  activity  sequencing  and  instructor  dialogue  content.  Figure  9  presents  a 
sample  script  for  enabling  objective  1.2. 2. 3  (Demonstrating  how  to  set  the  Back  Light  Timeout). 


Back  Light  Timeout 

014 

Objective 

Helpful  Content  for  the  Instructor 

DO  Review  basic  information  odout 

Enabhrg  Objective.  1 223 

setting  the  Bock  Light  Timeout  found 

Demonstrate  how  to  set  the  BackLight 
Timeout  and  share  stories  related  tothis 
functionality. 

The  LCD  Screen  hasa  lamp  (or  backlight).  This  allows  you  to 
view  the  screen  information  inthe  dark.  The  intensity  of  the 
ba  ckl  ighti  ng  ca  n  be  set  throug  h  the  G 1  oba  1  prog  ra  m  m  i  ng 

in  helpful  content.  Pol  low  the  1 
screen  activity  sequence  while 
onswering  questions. 

Screen  Activity  Sequence: 

menu. 

SAY:  Now,  it's  your  turn  to  set  the 
BLTimeout.  If  you  need  any 
assistance  during  your  activity,  use 
your  chat  function.  Also,  access  your 
knowledge  database  and  type  in  the 
KEY  phrase  "BLTimeout"  to  learn 

(1  sequence) 

1 

1.  Focus  o«i  key  pad  and  LCD  screen . 
i  2.  Press  ALT  and  Mode  at  the  same  time. 

1  3.  Display  the  MAIN  menu  (with  Program 

The  display  backlighting  is  turned  on  and  off  by  pressing  the 

ALT  and  ENT  key.  This  action  refers  to  LAMP  on  the  Keypad. 

The  display  backlighting  is  turned  off  by  the  same  action, 
pressing  the  ALT  and  ENT  key. 

htghfcghted) 

4.  Press  ENT. 

The  purpose  of  the  BLTimeout  is  to  turn  off  the  screen  lamp. 

more. 

1  5.  Use  Arrow  keys  to  scral  through 

Programming  options  until  Global  is 
highlighted. 

or  backlight,  after  a  specified  amount  of  time.  The  BLTimeout 

ACTION:*  User  is  asked  to  select 

can  be  setfor  10  seconds  to  40  seconds  in  ten  second 

20  Seconds  for  BLTimeout. 

6.  Press  ENT. 

7.  Display  the  Global  Options  :TX  Timeout  0L 
Timeout  {which  is  higHighted),  Set  Clock, 

MIC  leveL 

intervals,  or  INF  (not  recommended). 

“correct  and  completed  -  indiatediAAth  a 

Avoid  setting  your  BLTimeout  to  INF,  as  this  will  drain  the 

green  light  on  instructor  screen,  otherwise 

3.  Press  ENT  {since  BL  Timeout  already 
highlighted) 

battery  and  could  com  promise  your  location  especially  if  your 

9.  Use  Arrow  keys  to  select 20  Sec 

10.  Press  ENT. 

radio  is  attached  to  you  while  you  are  moving. 

Figure  9:  Virtual  classroom  script. 

Collaborative  scenario.  The  objective  of  the  collaborative  scenario  was  slightly 
different  from  that  of  the  mobile  application  and  the  virtual  classroom.  Specifically,  the  purpose 
of  the  collaborative  scenario  was  to  assess  students’  ability  to  use  and  manipulate  the  radio. 
Therefore  new  material  was  not  presented.  Therefore,  the  entire  scenario  essentially  functioned 
as  an  assessment.  Below  is  an  overview  of  the  development  process,  which  included: 

•  Developing  the  collaborative  scenario  concept, 

•  Identifying  subordinate  objectives, 

•  Collecting  critical  incidents, 

•  Creating  vignettes,  and 

•  Developing  a  collaborative  scenario  technical  platform. 

A  description  of  the  scoring  approach  is  provided  in  greater  detail  later  in  the  report; 
specifically  in  the  section  that  describes  the  methodology  for  creating  the  assessments. 
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Collaborative  scenario  concept.  Because  use  of  the  JEM  radio  use  is  not  an  inherently 
team  task,  it  was  necessary  to  create  an  underlying  concept  for  the  collaborative  scenario  before 
developing  any  content.  The  first  step  in  developing  the  collaborative  scenario  was  to  determine 
an  overarching  concept  or  framework  delineating  the  nature  of  the  exercise  and  the  pattern  of 
player  interaction  (i.e.,  how  they  would  collaborate).  After  consulting  SMEs  and  SGLs  about 
use  of  the  radio  in  operational  settings,  the  current  scenario-based  concept  was  developed.  The 
concept  involved  three  players  working  together  to  complete  a  series  of  tasks  using  the  radio  as 
they  progressed  through  a  virtual  “day  in  the  life”  of  a  signal  support  systems  specialist. 

There  were  several  advantages  to  this  design  concept.  First,  the  “day  in  the  life” 
approach  offered  an  opportunity  for  learners  to  apply  what  they  had  learned  during  the  mobile 
and  virtual  classroom  trainings,  preparing  them  for  knowledge  and  skill  transfer  in  the  real 
world.  This  type  of  scenario  also  added  a  decision-making  dimension  (i.e.,  “when  to  do”)  in 
addition  to  the  knowledge  and  skill  dimension  (i.e.,  “how  to  do”).  This  dimension  made  it  an 
ideal  approach  for  evaluating  performance  on  tasks  that  required  critical  thinking  and  decision¬ 
making.  Second,  the  training  technology  could  be  leveraged  to  provide  a  simulated  operational, 
high-fidelity,  contextualized  training  environment  that  is  more  engaging  than  lecture  based 
methods;  a  characteristic  that  is  likely  to  lead  to  more  effective  training  and  transfer  (Noe,  Tews, 
&  McConnell  Dachner,  2010).  Third,  the  setup  of  the  scenario  provided  both  structural  and 
emergent  components  of  collaboration.  The  structural  component  (i.e.,  the  radio  being  passed 
from  one  player  to  the  next)  ensured  collaboration  because  all  players  needed  to  succeed  in  their 
scenarios  for  the  team  to  accomplish  the  overall  goal.  The  emergent  component  (i.e.,  how  the 
players  will  work  together  over  the  open  communication  channel)  reflected  natural  collective 
performance  processes. 

Subordinate  objectives.  The  collaborative  scenario  was  designed  to  allow  the  learner  to 
demonstrate  operation  and  troubleshooting  of  the  JEM  as  required,  and  to  evaluate  other  team 
members’  use  of  the  JEM  through  guidance,  collaboration,  and  after  action  review  (AAR). 
Content  for  the  collaborative  scenario  was  developed  using  the  following  sources: 

•  JEM  Technical  Manual, 

•  JEM  classroom  presentation  slides  used  in  the  ALC  at  Fort  Gordon,  and 

•  SME  interview  data  collected  during  data  collection  Trips  2,  3,  and  4. 

Table  6  contains  a  listing  of  the  subordinate  objectives  identified  for  the  collaborative 
scenario. 
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Table  6 


Virtual  Classroom  Platform  Subordinate  Objectives 


Subordinate  Objective 

Description 

1.3.1 

Identify  unusual  conditions  and  estimate  the  impact  these  conditions 
will  have  on  the  proper  use  of  the  radio  and  formulate  how  to 
accommodate  the  unusual  condition. 

1.3.2 

Configure  and  manipulate  the  JEM  properly  and  maintain 
communications  in  a  scenario  of  usual  conditions  while 
troubleshooting  as  required. 

1.3.3 

Execute  guided  practice  and  provide  feedback  to  reach  a  successful 
conclusion  of  established  communications,  given  a  scenario  of  usual 
conditions. 

1.3.4 

Configure  and  manipulate  the  JEM  properly  and  maintain 
communications  in  a  scenario  of  unusual  conditions  while 
troubleshooting  and  adapting  as  required. 

1.3.5 

Execute  guided  practice  and  provide  feedback  to  reach  a  successful 
conclusion  of  established  communications,  given  a  scenario  of  unusual 
conditions. 

The  ISD  Map  shows  how  these  five  subordinate  objectives  were  decomposed  into 
specific  enabling  objectives  such  as  "Describe  the  circumstances  of  an  ordinance-ridden  area  and 
what  impact  this  could  have  on  radio  use.”  The  objectives  were  then  leveraged  for  content 
development  in  the  collaborative  scenario  along  with  information  collected  from  critical 
incidents. 

Critical  incidents.  Critical  incidents  are  examples  of  actual  behavior  that  illustrate 
outstanding  or  unacceptable  levels  of  job  performance  (Brannick,  Levine,  &  Morgeson,  2007). 
The  purpose  of  collecting  critical  incidents  for  this  effort  was  to  provide  context  for  the  learning 
objectives  identified  so  that  the  scenario  could  be  as  real  as  possible.  Once  the  concept  for  the 
collaborative  scenario  was  determined,  SMEs  were  interviewed  at  Fort  Gordon  to  collect  critical 
incidents.  During  these  meetings  two  SMEs  described  detailed  scenarios  that  involved  using  the 
JEM  radio  in  an  operational  context.  The  questioning  technique  used  also  allowed  the  team  to 
establish  requirements  for  appropriate  art  assets,  virtual  environment  components,  player  actions, 
and  non-player  character  (NPC)  actions.  Based  on  the  information  collected,  10  initial  vignettes 
were  confirmed  within  one  scenario  story  of  a  Personnel  Security  Detachment  (PSD)  mission 
where  a  Colonel  visits  a  local  village  to  check  on  the  progress  of  a  new  school  under 
construction.  Various  tasks  related  to  the  use  of  the  radio  were  required  of  Soldiers 
accompanying  the  Colonel. 

Vignettes.  Building  on  the  critical  incidents  collected,  detailed  vignettes  for  the 
collaborative  scenario  were  developed.  Several  key  considerations  guided  the  development 
process.  First,  the  design  team  ensured  the  situation  players  encountered  would  not  result  in 
repeated  task  execution;  the  tasks  required  in  the  vignettes  were  designed  to  be  mostly 
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independent  to  avoid  over-penalizing  the  same  mistake  (e.g.,  performance  on  one  task  would  not 
affect  performance  on  another  task).  Second,  the  subordinate  objectives  from  the  ISD  Map  were 
referred  to  in  order  to  ensure  that  a  variety  of  tasks  were  covered  throughout  the  vignettes. 
Finally,  the  vignettes  were  written  so  that  the  assessments  strictly  adhered  to  the  learning 
objectives  and  training  content  covered  in  the  ISD  map;  they  did  not  involve  any  tactical 
decision-making  or  extraneous  criterion  elements. 

At  the  end  of  the  development  process,  seven  fully  developed  vignettes  were  derived  (see 
Appendix  B).  These  vignettes  were  developed  using  a  storyboard  method  that  documented  each 
vignette  with  all  of  its  components  in  order  of  appearance.  For  each  vignette,  trainees  are  first 
presented  with  a  video  clip  that  provides  each  player  with  an  overview  of  the  task  and  setting  for 
the  radio.  After  viewing  the  video,  trainees  are  prompted  to  answer  one  or  two  individual-level 
knowledge  check  questions  (see  Assessments  section  to  follow).  The  content  of  the  vignettes 
was  validated  by  SMEs  and  SGLs  at  Fort  Gordon.  Specifically,  SMEs  provided  the  development 
team  with  feedback  with  regard  to  the  content  and  face  validity  of  each  vignette.  This 
information  was  used  to  revise  each  vignette  in  the  collaborative  scenario. 

Collaborative  scenario  technical  platform.  The  vignette  screenplays  served  as  a 
functional  specification  in  the  development  of  the  animated  sequences  and  assessments  for  the 
collaborative  scenario.  The  technologies  used  to  create  the  Virtual  Classroom  (Unity  Master 
Server,  Unity  Multiplayer  Networking  API,  TeamSpeak  VIOP  SDK,  and  Unity3D)  and  the 
virtual  radio,  were  also  employed  in  the  collaborative  scenario. 

Assessment  Development 

Concurrent  with  the  training  technology  development,  an  assessment  team,  consisting  of 
research  psychologists  created  a  suite  of  training  assessments  that  included:  a  CAT;  interim 
assessments  for  the  mobile  training  and  the  virtual  classroom;  and  individual  and  group  level 
assessments  for  the  collaborative  scenario.  The  provisional  guidelines  outlined  in  Brusso  et  al. 
(2013)  were  used  to  direct  the  development  of  the  prototype  assessments.  For  example, 
assessments  were  embedded  within  each  training  platform  to  maximize  training  efficiency;  and 
frequent  testing  was  employed  throughout  the  training  prototypes  as  a  means  of  reinforcing 
learning.  The  follow  sections  contain  a  description  of  how  each  assessment  type  was  developed. 

Assessment  data,  along  with  student  profiles,  were  stored  in  a  prototype  implementation 
of  the  SCALE  architecture  (Mangold  et  al.,  2012).  This  solution  provided  a  web-based  interface 
that  allowed  administrators  to  create  and  authenticate  student  accounts  for  each  of  the  training 
modalities  (mobile,  virtual  classroom,  and  collaborative  scenario).  The  SCALE  architecture  also 
provided  a  means  to  record  and  store  interaction  data  from  each  modality.  The  prototype  was 
built  as  an  extension  of  a  Drupal  content  management  system  that  leverages  PHP  (Hypertext 
Preprocessor)  code  and  uses  a  MySQL  (My  Structured  Query  Language)  database  to  provide 
persistent  storage  of  interaction  and  assessment  data. 

Computer  adaptive  testing.  To  complement  the  mobile  device  training  and  the  virtual 
classroom  modules,  and  to  measure  learning  between  training  modalities,  the  assessment  team 
developed  a  CAT;  it  was  specifically  designed  to  serve  as  the  pre-  and  post-test  at  the  beginning 
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and  conclusion  of  the  mobile  training,  as  well  as  after  the  virtual  classroom  training.  The 
assessment  team  employed  item  response  theory  (IRT)  and  adaptive  testing  principles  as  the  base 
psychometric  technology  for  the  CAT.  IRT  does  not  assume  that  every  item  is  equal  with  regard 
to  assessing  student  ability.  Rather,  it  assumes  that  measurement  can  become  more  precise  by 
accounting  for  several  item  characteristics  and  their  relationship  with  the  trait  being  measured. 
With  dichotomously  scored  multiple  choice  items,  up  to  three  IRT  parameters  can  be  used  in 
adaptive  testing:  item  difficulty,  item  discrimination,  and  a  “guessing”  parameter.  In  the  one- 
parameter  logistic  model  (1PLM;  Rasch,  1960)  item  difficulty  is  the  only  item  property  used  to 
identify  the  best  items  to  present  to  an  examinee  for  ability  and  standard  error  estimation.  In  the 
two-parameter  logistic  model  (2PLM;  Bimbaum,  1968)  item  discrimination  is  also  considered. 
The  discrimination  parameter  reflects  an  item’s  measurement  sensitivity  at  different  ability 
levels.  The  three-parameter  logistic  model  (3PLM;  Birnbaum,  1968)  includes  an  additional 
parameter,  which  relates  to  the  likelihood  of  a  low-ability  examinee  getting  an  item  correct  due 
to  chance  factors  such  as  guessing. 

Depending  on  the  IRT  model  desired,  different  sample  sizes  are  required  for  parameter 
estimation.  The  3PLM  typically  provides  good  fit  to  data  for  knowledge  and  skills  based 
assessments,  but  it  requires  a  large  sample  size  (1000  or  more).  As  such,  many  CAT  programs 
initially  use  simpler  alternatives,  such  as  the  2PLM  or  the  1PLM  (Hambleton,  Swaminathan,  & 
Rogers,  1991).  With  the  1PLM,  samples  of  150  for  item  parameter  estimation  are  not 
uncommon. 

For  this  effort,  the  CAT  was  based  on  the  1PLM.  Because  there  were  no  pre-existing 
data  on  the  difficulty  of  the  test  items,  appropriately  scaled  SME  ratings  of  item  difficulty  were 
used  in  place  of  actual  IRT  item  difficulty  parameter  estimates  (see  section  on  item  validation). 
This  method  has  been  used  previously  by  researchers  in  developing  CAT.  For  instance,  Stark, 
Chernyshenko,  and  Guenole  (2011),  found  that  SME-based  item  difficulty  ratings  were  strongly 
correlated  with  true  proficiency  scores  (+.90),  even  when  the  SME  ratings  correlated  only 
moderately  (.60)  with  actual  IRT  difficulty  parameters.  This  finding,  coupled  with  the 
developmental  purpose  of  this  current  training  assessment  prototype,  suggests  SME  difficulty 
ratings  were  a  viable  and  useful  means  for  estimating  item  difficulty,  at  least  until  sufficient 
empirical  data  have  been  collected  to  calculate  the  more  traditional  maximum  likelihood  IRT 
parameter  estimation. 

One  advantage  of  CAT  is  that  scoring  does  not  require  trainees  to  receive  the  same 
sequence  of  items;  instead  a  subset  of  items  is  chosen  from  an  item  pool  based  on  estimated 
ability  level.  Consequently,  trainees  can  take  an  assessment  more  than  once  without  seeing  the 
same  exact  items,  which  helps  reduce  recency  effects  and  maintains  a  sufficient  level  of 
challenge.  In  addition,  CAT  maximizes  testing  time  by  identifying  the  most  diagnostic  items  for 
each  trainee,  and  administering  these  items  based  on  an  estimate  of  learner  ability.  Research  has 
shown  that  adaptive  testing  can  reduce  test  length  by  approximately  50%  without  losses  in 
measurement  precision  (Weiss  &  Kingsbury,  1984).  The  detailed  development  procedures  for 
the  CAT  are  presented  in  the  following  subsections. 

The  assessment  team  used  a  series  of  steps  to  design  the  CAT.  This  process  included 
developing  a  test  plan  based  on  the  ISD  Map,  drafting  test  items,  reviewing  and  revising  test 
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items,  and  collecting  SME  ratings  of  the  items.  Revisions  were  incorporated  iteratively 
throughout  the  process  to  ensure  the  items  accurately  measured  the  knowledge  mapped  in  the 
training  content. 

Test  plan.  The  CAT  item  pool  was  designed  to  vary  sufficiently  in  difficulty  to 
accurately  assess  individuals  across  all  proficiency  levels  in  every  content  domain  relevant  to  the 
JEM  radio  training.  To  achieve  this  goal,  the  assessment  team  first  reviewed  the  ISD  Map  and, 
based  on  the  subordinate  learning  objectives,  synthesized  the  content  into  four  main  subject 
domains:  Functionality  /  Basic  Procedures,  Complex  Procedures,  Terminology  and 
Specifications,  and  Troubleshooting.  Next  a  test  plan  for  each  domain,  based  on  the  enabling 
objectives  in  the  ISD  Map,  was  created  to  guide  the  item  development  process.  For  each  content 
domain,  the  test  plan  included  a  guide  on  how  to  develop  items  that  varied  in  difficulty  level 
(easy,  medium,  and  difficult).  The  proposed  difficulty  stratification  in  the  test  plan  was  intended 
to  provide  a  roadmap  for  item  writers  to  ensure  sufficient  variance  in  difficulty  of  the  draft  items. 
Once  draft  items  were  developed,  the  actual  item  difficulty  parameters  were  estimated 
empirically  using  SME  ratings  on  a  seven-point  Likert  scale  (see  section  on  item  validation). 

Test  items.  To  begin  the  item-writing  process,  the  assessment  team  first  became  familiar 
with  the  JEM  radio  training  content.  Each  team  member  attended  a  training  session  during 
which  members  were  provided  a  set  of  uniform  guidelines,  including  the  test  plan,  to  ensure 
consistency  in  the  quality  of  items  drafted.  Examples  of  items  at  various  difficulty  levels  were 
provided  to  assist  them  with  the  item  development  process.  Then  each  item  writer  was  assigned 
a  portion  of  the  test  plan  for  which  to  develop  items.  The  majority  of  the  item-writing 
assignments  were  distributed  among  three  key  item  writers  with  two  additional  item  writers 
serving  as  reviewers  and  writing  additional  items  as  needed.  A  Microsoft  Access  database  was 
constructed  to  store  and  organize  all  the  draft  items.  For  each  item  entered,  the  associated 
content  domain,  enabling  objective  and  projected  item  difficulty  were  documented. 

Throughout  the  item  development  process,  the  assessment  team  convened  regularly  to 
report  progress  and  discuss  any  issues  encountered.  Once  all  draft  items  were  created,  an 
internal  review  process  was  conducted  to  examine  the  language,  clarity,  and  relevance  of  the 
items.  As  part  of  this  internal  review,  all  items  were  examined  by  internal  SMEs  for  accuracy 
and  relevance.  Items  that  were  ambiguous  or  inaccurate  were  revised  accordingly. 

Item  Validation.  Next,  the  draft  item  bank  was  presented  to  SMEs  at  Fort  Gordon  for 
feedback  as  part  of  the  item  content  validation  process.  During  this  workshop,  SMEs  were  asked 
to  scrutinize  each  draft  item  and  provide  edits  when  necessary.  At  the  beginning  of  the 
workshop,  the  SMEs  received  a  brief  introduction  on  the  purpose  of  the  validation  process.  They 
also  received  a  list  of  questions  to  consider  during  the  review,  such  as: 

•  Is  the  item  stem  written  clearly  (i.e.,  is  the  question  being  asked  easily 
understood)? 

•  Is  the  terminology  pertaining  to  the  JEM  radio  used  correctly  in  the  item? 

•  Is  there  only  one  correct  answer  to  the  question? 

•  Are  the  incorrect  response  options  effective? 

•  Is  there  a  better  answer  to  the  question  that  is  not  one  of  the  response  options? 


22 


•  Is  the  intended  difficulty  level  for  the  item  appropriate? 

In  addition,  SMEs  were  asked  to  review  all  the  items  vis-a-vis  the  ISD  Map  to:  (a) 
determine  whether  the  item  pool  adequately  covered  all  content  domains  in  the  training,  and  (b) 
identify  any  potential  gaps  or  missing  areas  where  new  items  needed  to  be  added.  Finally,  as 
part  of  the  validation  process,  SMEs  were  asked  to  estimate  the  level  of  difficulty  for  each  item. 
Specifically,  they  were  asked  to  provide  ratings  on  the  relative  level  of  difficulty  of  each  item  on 
a  seven-point  Likert  scale,  from  very  low  to  very  high.  The  scale’s  answers  were  as  follows:  1  = 
very  low;  2  =  moderately  low;  3  =  slightly  low;  4  =  average;  5  =  slightly  high;  6  =  moderately 
high;  and  7  =  very  high.  SMEs  first  independently  reviewed  the  items  to  provide  individual  edits 
and  ratings.  Once  they  finished  the  independent  review,  they  were  debriefed,  and  given  an 
opportunity  to  discuss  their  feedback  in  more  detail. 

Upon  conclusion  of  the  fourth  data  collection,  all  SME  feedback  was  reviewed  and 
incorporated,  and  items  deemed  unclear  or  irrelevant  were  either  revised  or  dropped.  A  fully 
developed  and  validated  pool  of  multiple  choice  items  was  then  completed  and  loaded  into  the 
CAT  technical  platform  for  implementation  in  the  training. 

CAT  technical  platform.  The  current  CAT  technical  platform  is  configured  to  support 
computer  adaptive  assessments  before  and  after  the  mobile  and  virtual  classroom  training 
modules.  Each  assessment  consists  of  12  items  drawn  from  a  183-item  pool  covering  the  four 
relevant  content  areas  (i.e.,  Functionality  /  Basic  Procedures,  Complex  Procedures,  Terminology 
and  Specifications,  and  Troubleshooting);  the  number  of  items  administered  for  each  content 
area  is  currently  set  to  three.  The  CAT  includes  text,  graphic,  and  audio-based  multiple-choice 
items.  Examples  of  audio  and  graphic -based  items  are  shown  in  Figures  10  and  11. 
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Figure  10:  Audio-based  CAT  item. 


Figure  11:  Graphic-based  CAT  item. 

The  current  set  of  three  assessments  proceeds  as  follows.  At  the  beginning  of  the  first 
assessment,  the  testing  platform  loads  assessment  specifications  consisting  of  the  total  number  of 
items  to  be  administered,  the  number  of  items  per  content  area,  and  the  available  item  pool. 

Next,  a  content  blueprint  is  created  by  randomly  ordering  the  desired  content  areas  for  test 
administration.  Specifically,  if  three  items  per  four  content  areas  are  specified,  the  testing 
platform  will  create  a  12-item  test  where  the  four  content  areas  appear  three  times  each  in  a 
random  order. 

Next,  an  examinee  is  assigned  a  provisional  assessment  score  of  zero  (0.0)  and  the  test 
begins  by  selecting  the  first  item  from  a  pre-specified  content  area.  Item  selection  is  adaptive 
throughout  the  whole  assessment.  It  calculates  information  values  (i.e.,  IRT  statistics  for 
measurement  precision)  at  each  examinee’s  current  score  for  all  available  items  in  that  content 
area  (based  on  the  one  parameter  logistic  IRT  model).  After  an  examinee  answers  the  first  item, 
his  or  her  score  is  calculated  via  IRT  expected  a  posteriori  (EAP)  estimation  and  the  next  item  is 
chosen  to  satisfy  both  content  and  information  requirements.  The  assessment  continues  until  all 
12  items  have  been  administered.  Then,  the  properties  of  administered  items,  examinee  answer 
choices,  scored  item  responses,  and  the  examinee’s  final  test  score  are  recorded  in  a  database. 
Subsequent  assessments  proceed  in  the  same  manner,  except  that  data  from  previous  tests  are 
used  to  prevent  repetition  of  items.  To  improve  efficiency,  rather  than  starting  each  test  with  an 
initial  trait  score  of  0.0,  the  final  score  on  the  most  recent  assessment  is  used.  This  essentially 
allows  the  examinee  to  “pick  up”  where  he  or  she  left  off. 
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The  adaptive  testing  platform  can  be  configured  to  accommodate  different  numbers  of 
assessments,  content  areas  within  each  assessment,  numbers  of  items  in  the  pool  representing 
each  content  domain,  and  different  test  lengths.  A  test  designer  can  also  modify  rules  for  item 
repetition  and  choose  alternative  starting  values  for  the  assessments. 

Checks  on  learning.  In  line  with  the  provisional  guidelines  from  Brusso  et  al.  (2014)  to 
employ  frequent  testing  to  reinforce  learning,  periodic  assessments  were  presented  to  Soldiers 
during  the  mobile  and  virtual  classroom  training  environments.  These  assessments  were  called 
“checks  on  learning”  to  reflect  terminology  currently  used  in  the  classroom.  Creation  and 
implementation  of  the  checks  on  learning  items  within  each  platform  is  discussed  next. 

Mobile  checks  on  learning.  A  bank  of  interim  assessment  items  was  created  for  each 
subordinate  objective  in  the  mobile  training.  The  items  were  grouped  into  nine  chunks  and 
administered  at  roughly  equal  intervals  throughout  the  training.  Each  chunk  only  focused  on 
content  covered  in  the  training  that  immediately  preceded  it  (since  the  last  check  on  learning 
assessment).  Learners  are  prevented  from  returning  to  previous  training  content  while  engaged 
in  a  check  on  learning  assessment,  but  are  able  to  revisit  material  after  completing  it.  The  check 
on  learning  assessments  served  multiple  purposes.  They  assessed  trainees  understanding  of  the 
previously  presented  training  material,  allowed  trainees  to  practice  applying  the  knowledge  and 
principles,  and  provided  trainees  with  feedback.  This  feedback  was  especially  relevant  in  the 
mobile  training  because  it  was  designed  to  be  completed  individually,  without  the  guidance  of  a 
live  instructor. 

The  first  step  in  drafting  the  interim  assessment  items  was  to  develop  eight  different  item 
formats.  The  different  formats  required  trainees  to  practice  applying  their  knowledge  in  varied 
and  unique  ways  and  utilized  the  available.  Table  7  lists  and  describes  each  item  format. 

The  assessment  team  then  developed  items  for  each  subordinate  objective  (roughly,  one 
item  per  enabling  objective).  Items  were  developed  to  utilize  all  of  the  different  formats 
although  some  were  used  more  than  others  (depending  on  which  formats  were  seen  as  the  best  fit 
to  assess  the  training  content).  Each  item  includes  response  options  that  could  be  selected  using 
the  tablets  interactive  touch-screen  interface.  To  submit  an  answer,  learners  click  the  “Submit” 
button  after  responding  to  an  item.  After  clicking  submit,  learners  receives  both  immediate 
corrective  text-based  feedback  on  their  response  (whether  it  was  correct  /  incorrect,  with  a  brief 
explanation  of  the  correct  answer)  and  auditory  feedback  (a  “ding”  with  a  green  check  to  indicate 
correct  or  a  buzz  with  a  red  “X”  to  indicate  incorrect).  If  a  learner  submits  an  incorrect  response 
to  an  item  in  the  virtual  JEM  programming  format,  the  process  to  continue  to  the  next  item  is 
slightly  different.  After  submitting  an  incorrect  response  to  a  virtual  JEM  programming  item,  the 
learner  cannot  continue  to  the  next  item  until  the  action  required  is  performed  correctly.  The 
feedback  presented  to  the  learner  provides  step-by-step  guidance  regarding  the  correct 
procedural  action.  After  following  these  steps,  the  learner  must  again  click  submit.  The  item  is 
still  scored  as  incorrect;  however,  this  procedure  aids  in  learning  by  having  the  learner  master  the 
procedure  before  moving  forward. 

After  drafting  a  complete  set  of  items  for  each  subordinate  objective,  these  items  were 
reviewed  by  SMEs  during  the  fourth  data  collection  trip.  SMEs  provided  input  on  the  accuracy 
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of  the  content  in  each  item,  as  well  as  the  accuracy  of  the  correct  answer  and  response  option 
distracters.  In  some  cases,  items  were  discarded  and  re-written  with  the  assistance  of  the  SMEs. 
Following  the  fourth  trip,  the  items  were  finalized  by  the  assessment  team.  The  check  on 
learning  activities  were  designed  to  require  students  to  answer  each  question  before  proceeding 
to  the  next  item  or  before  advancing  to  the  next  lesson.  A  timer  was  added  to  each  item,  such 
that  if  the  learner  remained  on  one  item  for  more  than  five  minutes,  time  expired,  and  the  learner 
is  forced  to  move  on  to  the  next  item.  The  five  minute  time  limit  was  chosen  because  it  is  not 
expected  to  impact  the  ability  of  even  the  slowest  test  taker  to  complete  the  item  correctly  (given 
their  content,  format,  and  difficulty).  However,  if  reached,  the  imposed  time  limit  will  move  the 
candidate  forward  so  that  the  check  on  learning  assessment  can  come  to  a  conclusion  within  a 
reasonable  amount  of  time. 

The  mobile  training  item  blueprint,  including  a  summary  of  the  items  by  check  on 
learning  assessment  and  subordinate  objective  is  provided  in  Table  8. 

Most  items  are  worth  one  point  each;  however,  several  items  include  multiple  parts  and 
are  therefore  worth  multiple  points.  Following  each  check  on  learning  activity,  the  learner  can 
review  a  summary  of  which  items  they  answered  correctly  and  incorrectly. 
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Table  7 


Check  on  Learning  Item  Formats  and  Descriptions 


Item  Format 

Description 

Example 

True  or  False 

Learner  clicks  on  'True'  or  'False'  in 
response  to  the  item. 

Indicate  whether  the  following  statement  is  true  or  false. 

Initial  synchronization  is  necessary  for  all  encrypted 
operations. 

a.  True 

b.  Lalse 

Matching 

Two  or  more  items  are  presented 
together  with  multiple  response 
options.  Each  item  has  only  one 
correct  answer.  When  an  option  is 
dragged,  it  should  disappear  from 
the  option  list. 

Drag  the  definition  on  the  right  to  the  associated  term  on  the 
left. 

1.  COMSEC 

2.  TRANSEC 

a.  Measures  taken  to  deny  unauthorized  persons  information 
derived  from  telecommunications. 

b.  Measures  taken  to  protect  transmissions  from  the 
interception  and  exploitation  by  means  other  than  crypto¬ 
analysis  (for  example,  "jamming"). 

Choose  all  that  apply 
-  Matching 

Two  or  more  items  are  presented 
together.  Response  options  are 
displayed  underneath.  Learner  can 
drag  option(s)  up  to  applicable  item. 
When  an  option  is  dragged,  the 
option  remains  below  so  it  could  be 
used  again  for  the  other  item.  Some 
items  include  images. 

Match  the  word  on  the  right  with  where  it  belongs  in  the 
acronym  on  the  left. 

1.  Multiband  Inter/Intra  T  Radio 

2.  Joint  T  Radio  System 

a.  Team 

b.  Technical 

c.  Tactical 

d.  Telecommunications 

Multiple  Choice 

Learner  selects  the  one  best  response 
from  a  list  of  options.  Some  items 
include  pictures  in  the  item  stem  or 
response  options. 

Choose  the  best  answer. 

Which  of  the  following  is  NOT  a  characteristic  of 

SINCGARS? 

a.  Single  channel 

b.  Frequency  hopping 

c.  VHF 

d.  UHF 
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Table  7  (continued) 

Check  on  Learning  Item  Formats  and  Descriptions 


Choose  all  that  apply 
-  Multiple  Choice 

Formatted  like  a  multiple-choice 
item,  except  the  learner  can  select  as 
many  options  as  desired.  Some 
items  include  images. 

Choose  all  that  apply. 

A  Soldier  needs  to  transfer  the  programming  of  one  JEM  to 
another.  He  has  successfully  connected  the  JEMs  with  the 
cloning  cable.  Which  of  the  following  can  he  expect  to  be 
transferred  during  cloning? 

a.  COMSEC  keys 

b.  Global  settings 

c.  Channel  settings 

d.  S  INC  GARS  loadsets 

e.  SINCGARS  NET  TIME 

f.  Group  settings 

e.  Real  time  clock  time 

Interactive  Multiple 
Choice 

Item  includes  a  bar  with  arrows  or  a 
movable  icon  that  the  learner  can 
slide  to  indicate  the  answer. 

Indicate  your  answer  on  the  ruler. 

What  is  the  BASIC  frequency  range  of  the  JEM?  Drag  the 
arrows  to  indicate  the  minimum  and  maximum  frequencies. 
[Ruler  marked  with:  0,  30,  60,  128,  256,  512,  724  MHz] 

Virtual  JEM  - 
Without 
programming 

Items  include  a  virtual  JEM.  Items 
require  the  learner  to  click  on  the 

JEM  to  identify  specified 
components,  to  drag  labels  to  the 
appropriate  location  on  the  JEM,  or 
to  connect  external  components  to 
the  JEM.  Items  do  not  require  use 
of  the  keypad  for  programming. 

[Show  virtual  JEM] 

Drag  the  name  of  the  control  to  its  location  on  the  JEM. 

a.  On  /  Off  switch 

b.  Channel  select  switch 

c.  Internal  speaker  microphone 

d.  Keypad 

e.  Mechanical  interlock 

Virtual  JEM  -  With 
programming 

Items  include  a  virtual  JEM  that  the 
learner  must  program  using  the 
keypad. 

Use  the  keypad  to  complete  each  of  the  following  actions. 

After  you  complete  each  action,  click  submit,  and  you  will  be 
prompted  to  perform  the  next  action. 

1.  Change  the  selected  group  to  Group  4  (G04). 

2.  Open  the  channel  scan  screen. 

3.  Turn  the  lamp  on. 
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Table  8 


Mobile  Training  Check  on  Learning  Item  Blueprint 


Check  on  Feaming  Assessment 

Subordinate  Objective 

Number  of  Items 

1 

1.1.1 

4 

2 

1.1.1 

6 

3 

1.1.1 

8 

4 

1.1.2(AB) 

7 

5 

1.1.2(AB) 

2 

1.1.2(C) 

7 

6 

1.1.2(C) 

8 

7 

1.1.3 

11 

8 

1.1.3 

6 

9 

1.1.4(A) 

7 

1.1.4(B) 

1 

1.1.4(C) 

2 

Total  Number  of  Items 

69 

Virtual  classroom  checks  on  learning.  Virtual  classroom  checks  on  learning  were 
developed  in  a  similar  fashion  to  those  developed  for  the  mobile  training.  The  assessments  used 
a  number  of  item  formats  that  required  students  to  apply  the  material  covered  in  the  previous 
modules  through  interactive  activities. 

Several  differences  between  the  virtual  classroom  and  mobile  training  checks  on  learning 
are  worth  noting.  First,  the  virtual  classroom  items  required  the  use  of  a  mouse  instead  of  a 
touchscreen  interface.  This  requirement  did  not  impact  developing  the  item  content,  although  it 
may  impact  the  learner  experience  when  completing  items  in  the  virtual  classroom  versus  the 
mobile  training. 

Another  difference  was  the  varying  goals  of  Terminal  Objectives  for  the  mobile  training 
and  virtual  classroom  training.  Whereas  the  mobile  training  had  a  stronger  focus  on  declarative 
knowledge,  the  virtual  classroom  had  a  greater  focus  on  demonstrating  that  knowledge. 
Therefore,  the  virtual  classroom  included  a  greater  ratio  of  virtual  JEM  programming  items.  As 
previously  stated,  the  alignment  of  objectives,  and  subsequently  training  content,  and  delivery 
method  are  a  result  of  the  match  between  the  content  and  the  affordances  of  the  training 
technology  (see  Bower,  2008).  Thus,  the  training  in  the  mobile  environment  was  more  heavily 
focused  on  declarative  knowledge,  whereas  the  virtual  classroom  focused  on  building  on  this 
declarative  knowledge.  This  progression  aligns  with  theoretical  models  of  learning  or  training 
progression  (e.g.,  Anderson’s  ACT*  theory;  see  Anderson  1987,  1996).  As  discussed  by 
Goldstein  and  Ford  (2002),  these  stages  specify  declarative  knowledge  (i.e.,  factual  knowledge) 
as  the  necessary  first  step  in  the  learning  process.  Following  the  retention  of  declarative 
knowledge,  trainees  engage  in  a  knowledge  compilation  stage,  a  transitional  stage  where  learners 
begin  to  progress  from  possessing  declarative  knowledge,  or  the  “what”,  to  eventually  possessing 
procedural  knowledge,  the  “how”  and  “when.”  Anderson’s  ACT  theory  specifies  that  the 
effective  knowledge  of  procedures  only  occurs  after  a  trainee  possesses  the  requisite  declarative 
knowledge  (Anderson,  1996). 


29 


A  third  difference  was  the  item  completion  time  limit.  All  items  in  the  mobile  training 
included  a  5  minute  limit,  whereas  some  in  the  virtual  classroom  were  given  a  10  minute  limit 
because  they  were  slightly  more  in-depth,  particularly  the  virtual  JEM  programming  items. 

Finally,  the  following  options  were  only  available  in  the  virtual  classroom:  the  instructor 
can  review  how  learners  are  performing  on  the  assessments,  and  choose  to  review  material  if  it 
seems  several  learners  are  not  acquiring  the  necessary  knowledge  and  skills.  After  a  check  on 
learning  assessment  is  complete,  learners  can  also  utilize  the  chat  feature  and  ask  the  instructor 
questions.  Although  these  capabilities  did  not  impact  the  approach  to  drafting  items,  they  may 
impact  the  degree  to  which  learners  feel  they  learned  from  the  assessments  in  the  virtual 
classroom  in  comparison  to  the  mobile  training.  Social  cognitive  theory  (see  Bandura,  1986) 
would  support  the  idea  that  creating  a  social  presence  for  learners  (i.e.,  interacting  with  peers  and 
instructors)  would  likely  have  a  positive  impact  on  learning,  or  at  least,  affective  reactions  to 
learning.  These  interactions  can  allow  learners  to  learn  from  each  other  and  to  be  more  engaged 
in  the  instruction.  In  support  of  this  notion,  Van  Tassel  and  Schmitz  (2013)  demonstrated  that 
instructor-learner  interactions  do  in  fact  impact  learner  assessments  of  learning  satisfaction. 
Further,  Wei,  Chen,  and  Kinshuk  (2012)  demonstrated  that  perceptions  of  learning  interactions 
in  an  online  classroom  (ex.  “I  often  discussed  learning  issues  with  others  in  the  online 
classroom”)  had  a  positive  impact  on  actual  learning  performance. 

As  with  the  mobile  assessment  items,  items  drafted  for  the  virtual  classroom  were 
reviewed  by  SMEs  during  the  fourth  data  collection  trip.  Based  on  SME  comments,  items  were 
revised,  and  item  content  was  finalized.  The  final  virtual  classroom  check  on  learning  item 
blueprint  is  presented  below  in  Table  9. 

Table  9 


Virtual  Classroom  Check  on  Learning  Item  Blueprint 


Check  on  Learning  Assessment 

Subordinate  Objective 

Number  of  Items 

1 

1.2.1 

11 

2 

1.2.2 

8 

3 

1.2.3 

6 

4 

1.2.4 

2 

5 

1.2.5 

3 

6 

1.2.7 

3 

7 

1.2.8 

5 

Total  Number  of  Items 

38 

Collaborative  scenario.  In  conjunction  with  the  collaborative  scenario  development,  the 
assessment  team  embedded  assessments  and  feedback  into  the  exercise  itself.  Two  main 
assessment  categories  for  the  collaborative  scenario  were  created:  individual-level  knowledge 
check  questions  and  team-level  radio  manipulation  tasks.  Individual-level  knowledge  check 
items  were  developed  as  single-answer  multiple-choice  questions  (see  Figure  12).  Each  player 
completed  these  individually  without  communication  with  other  players  (i.e.,  the  chat  function  is 
disabled  until  everyone  completes  the  knowledge  check  questions).  All  radio  manipulation  tasks 
were  scored  at  a  team  level.  These  tasks  were  developed  as  a  series  of  steps  players  carried  out 
to  successfully  complete  a  vignette  (see  Figure  13).  In  each  vignette,  only  one  player  possessed 
control  of  the  simulated  radio  for  that  vignette.  However,  the  other  two  players  on  the  team  were 
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able  to  provide  input  and  collaborate  through  VoIP,  so  the  players  were  scored  as  a  team.  The 
final  score  was  determined  by  the  total  number  of  correct  steps  taken  by  members  of  the  team. 
Control  of  the  radio  was  rotated  from  player  to  player  as  the  team  progressed  through  the 
scenario. 

Assessment  items  were  drafted  in  conjunction  with  vignettes  using  the  vignette  story 
board  process  for  creation.  Following  a  short  video  presented  at  the  beginning  of  each  vignette 
learners  are  prompted  to  answer  one  or  two  individual-level  knowledge  check  questions.  Once 
all  knowledge  check  questions  are  completed,  the  player  assigned  as  the  active  player  for  the 
vignette  is  granted  control  of  the  simulated  radio  while  the  other  players  watch  the  active 
player’s  actions  live.  Some  radio  manipulation  tasks  also  involve  multiple-choice  questions  that 
the  active  player  must  answer  correctly  in  order  to  proceed.  For  example,  to  connect  or 
disconnect  parts  of  the  radio,  a  question  prompts  the  player  to  select  the  correct  physical  motion. 
These  process  questions  are  not  scored  separately  and  the  player  may  make  repeated  attempts. 
The  radio  manipulation  tasks  are  scored  based  on  whether  the  overall  goal  of  the  task  is 
successfully  achieved  within  the  time  limit.  The  task  also  included  animated  demonstration- 
based  feedback  videos  that  show  how  to  properly  perform  the  task  if  it  is  not  completed 
successfully. 

As  with  the  vignette  validation  for  the  collaborative  scenario,  SMEs  were  provided  with 
vignette  assessment  items,  including  both  the  vignettes  and  items.  Then  the  flow  of  the  vignette 
was  described  and  their  input  was  solicited  with  regard  to  the  realism  and  feasibility  of  the 
vignettes,  and  the  individual-level  knowledge  check  assessment  items.  The  collaborative 
scenario  content  was  revised  and  finalized  based  on  SME  feedback. 


Your  convoy  needs  to  assist 
another  Battalion  within  a  close 
distance.  When  you  are  about  half 
a  kilometer  away,  you  can't 
communicate  through  your  radio 
but  when  you  look  at  the  radio 
screen,  everything  seems 
functional. 


#  You  are  the  active  learner 


Answer  the  following: 

What  is  the  most  likely  reason  why 
no  one  is  able  to  communicate 
through  the  radio? 

I  I  The  battery  is  not  properly 
—  connected. 

I  I  The  radios  are  set  to  different 

I* 1—1  CTCSS  tones. 

|  The  side  connector  is  enabled. 

I  I  Your  external  mic  is 
—  malfunctioning 


Figure  12:  Collaborative  scenario:  Individual  assessment. 
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H  ^  E SMI  Voice 

Prepare  your  radio  for  the  mission  Press  "V1 

by  completing  all  the  necessary 

start-up  procedures.  Perform 

comms  check.  Clone  a  radio.  You 

have  30  minutes. 

"  to  pull  up  Vignette  Debug  Menu 

"Right-mouse"  button  to  rotate  the  radio 

"Mouse-wheel"  to  zoom  in  and  out  of  radio 

Your  instructions: 

■  Enter  Julian  Date  (32.08.00) 

-  Perform  comms  check. 

*  Clone  a  radio. 

,  > 

MENU  SCAM  >  OCK 

▼  A 

I 

THALES 

» 

Figure  13:  Collaborative  scenario:  Team  assessment. 

Method 

After  completing  development  of  full  versions  of  the  mobile  training,  virtual  classroom, 
and  collaborative  scenario,  a  beta  test  was  conducted  to  identify  any  issues  with  the  training 
content.  The  beta  test  was  an  opportunity  for  Soldiers  (most  with  prior  experience  using  the 
JEM)  to  review  the  training  materials  and  provide  feedback  on  the  content,  appearance  /  features, 
functionality,  and  organization  of  the  training  and  assessments  and  to  provide  critical  feedback 
on  the  usability  of  the  training  modalities. 

Participants 

Participants  in  the  beta  test  were  recruited  by  the  instructors  at  Fort  Gordon.  A  total  of  21 
Soldiers  participated  and  each  one  completed  training  on  at  least  one  of  the  three  modalities  (i.e., 
mobile,  virtual,  or  collaborative);  13  completed  two  modalities.  In  total,  the  mobile  training  was 
completed  by  18  Soldiers,  the  virtual  classroom  was  completed  by  seven  Soldiers,  and  the 
collaborative  scenario  was  completed  by  nine  Soldiers.  The  Soldiers  had  an  average  tenure  of  10 
years  in  the  military,  an  average  of  one  year  in  their  current  rank,  and  had  an  average  of  3.4 
deployments.  The  ranks  of  the  participants  were  split  between  Sergeant  First  Class  (SFC)  ( n  =  9) 
and  Staff  Sergeant  (SSG)  ( n  =  12).  Participants’  MOS  varied  between  25U  (Signal  Support 
Systems  Specialist;  n  =  12),  25C  (Radio  Operator/Maintainer;  n  =  7),  25B  (Information 
Technology  Specialist;  n  =  1),  and  25M  (Multimedia  Illustrator;  n  =  1). 

Procedure 

Participants  completed  the  beta  test  in  groups  of  three  in  two  locations.  They  were 
provided  with  either  iPads  or  desktop  computers,  with  the  group  provided  iPads  completing  the 
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mobile  training  and  those  with  desktops  completing  either  the  virtual  classroom  or  collaborative 
scenario. 

Upon  arriving  at  the  testing  location,  participants  were  presented  with  a  brief  introduction 
to  the  project  and  the  purpose  of  the  beta  test,  followed  by  an  explanation  of  the  tasks  they  were 
to  complete.  Participants  were  then  asked  to  sign  an  informed  consent.  After  submitting  the 
signed  informed  consent,  participants  received  an  information  packet  (see  Appendix  C)  that 
included  an  overview  of  each  training  prototype  and  user  feedback  questions  that  were  to  be 
completed.  These  questions  were  written  to  capture  participants  overall  satisfaction  with  the 
training  prototypes,  as  well  as  collect  feedback  regarding  technical  issues,  content,  instructional 
design,  and  overall  usability  of  the  training.  Participants  were  provided  with  instructions  to  use 
the  feedback  form  to  record  any  issues  observed  during  the  trainings  and  /  or  assessments.  Upon 
beginning  each  training  modality,  participants  logged-in  the  training  platform  using  a  unique 
username  and  password.  Once  logged-in,  participants  were  given  a  brief  overview  of  how  to 
interact  with  the  technology  (i.e.,  how  to  use  the  mobile  device)  and  associated  equipment  (e.g., 
headsets).  Then  participants  completed  their  respective  training  modality.  During  all  training 
modalities,  researchers  recorded  notes  based  on  participant  feedback.  Upon  finishing, 
participants  completed  an  online  training  reaction  questionnaire  to  record  thoughts  and  opinions 
about  the  particular  training  modality  (Appendix  D).  Participants  were  instructed  to  read  each 
question  carefully  and  respond  using  an  agreement  scale  ranging  from  1  ( strongly  disagree)  to  5 
(, strongly  agree).  Upon  completion,  participants  were  asked  several  interview  questions 
regarding  their  satisfaction  with  the  training.  After  answering  these  questions,  participants  were 
dismissed. 

Mobile  training.  After  receiving  the  overview  information  regarding  the  mobile 
platform  and  successfully  logging-in  to  the  training  platform,  participants  were  given  a  brief 
overview  of  the  functionality  of  the  technology  (i.e.,  the  iPad).  Once  participants  were  familiar 
with  the  technology,  they  proceeded  to  take  the  computer  adaptive  pre-test  via  the  iPad. 
Following  the  completion  of  the  pre-test,  participants  then  proceeded  to  the  training  content. 
During  this  time,  participants  navigated  through  the  training  modules  and  completed  checks  on 
learning  when  prompted.  At  the  conclusion  of  the  training,  participants  completed  the  computer 
adaptive  post-test  on  the  iPad.  Then  participants  completed  a  reaction  questionnaire  online. 
Participants  were  then  asked  the  following  four  questions  listed  at  the  end  of  Appendix  C: 

•  Did  you  find  the  trainings  more  or  less  helpful  than  traditional  classroom  learning? 

•  Did  you  find  the  technology  to  be  easy  to  understand  and  use? 

•  What,  if  anything,  would  you  change  about  the  trainings  and  assessments? 

•  Please  provide  any  additional  comments  you  have  about  the  trainings. 

The  debriefing  process  was  semi- structured  to  allow  participants  to  expand  upon  their  answers 
with  comments  documented  on  their  feedback  questionnaire.  Upon  completion  of  the  debriefing 
participants  were  dismissed. 

Virtual  classroom.  Once  participants  were  logged  in  to  the  virtual  classroom  via  a 
desktop  computer,  they  were  allowed  time  to  familiarize  themselves  with  the  technology.  Once 
familiar,  participants  completed  the  computer  adaptive  pre-test.  Following  the  completion  of  the 
pre-test,  participants  then  proceeded  to  the  training  content.  In  the  virtual  classroom,  a 
researcher  served  the  role  of  classroom  instructor  /  facilitator.  During  this  training,  participants 
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navigated  through  the  training  modules,  completed  checks  on  learning  when  prompted,  and 
interacted  with  each  other  and  the  instructor  via  headset.  At  the  conclusion  of  the  training, 
participants  completed  the  computer  adaptive  post-test  within  the  virtual  classroom  environment 
as  well  as  an  online  questionnaire  concerning  their  reactions  to  the  training.  Participants  were 
then  debriefed  with  the  four  questions  listed  above. 

Collaborative  scenario.  Following  the  general  training  overview  and  log-in  creation, 
participants  in  the  collaborative  scenario  completed  a  pre-recorded  tutorial  that  explained  the 
game  features.  Here,  participants  were  familiarized  with  the  training  content,  technology 
controls,  and  goals  and  instructions  for  the  individual  and  team  assessments.  After  completing 
the  tutorial,  participants  engaged  in  the  collaborative  scenario  and  proceeded  until  they 
completed  all  the  vignettes  and  associated  assessments.  As  with  the  previous  modalities, 
participants  completed  a  training  reactions  online  questionnaire  at  the  conclusion  of  the  training. 
Participants  were  then  debriefed  with  the  same  four  questions  asked  after  the  mobile  and  virtual 
classroom  (see  above). 


Results 

Responses  to  the  online  learner  reaction  survey  were  analyzed  to  better  understand  user 
perceptions  of  the  prototype  training  and  identify  any  potential  areas  for  improvement. 
Percentages  of  participants  that  were  favorable  (i.e.,  agreeing  or  strongly  agreeing  on  positive 
statements),  neutral  (i.e.,  neither  agreeing  nor  disagreeing  on  positive  statements),  and 
unfavorable  (i.e.,  disagreeing  or  strongly  disagreeing  on  positive  statements)  were  computed  for 
each  item.  Then  the  item  scores  were  aggregated  to  the  dimension-level  by  calculating  an 
average  score  across  all  items  within  that  dimension.  Results  for  each  topic  area  on  the  survey 
are  summarized  below;  they  should  be  interpreted  as  preliminary  because  this  was  not  a  full 
training  evaluation. 

Technology  Usability 

In  the  technology  section  of  the  survey,  participants  were  asked  to  rate  five  items  related 
to  ease  of  use,  accessibility,  and  functionality  of  the  technology  interface.  Summary  statistics  on 
all  items  in  this  section  are  presented  in  Table  10.  As  shown,  over  80%  of  participants  indicated 
favorable  attitudes  on  technology  for  the  mobile  and  virtual  classroom  modalities  across  all 
questions  on  this  topic.  Attitudes  were  less  favorable  for  the  collaborative  scenario  modality, 
with  slightly  over  60%  of  participants  indicating  favorable  reactions  to  collaborative  technology 
across  all  items. 
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Table  10 


Percentages  of  Favorable  Ratings  on  Technology  Usability  by  Training  Modality 


Technology 

Overall 
(N  =  34) 

Mobile 
(N  =  18) 

Virtual 
Classroom 
(N  =  7) 

Collaborative 
Scenario 
(N  =  9) 

1. 

The  technology  interface  was  easy  to 
use. 

82% 

89% 

100% 

57% 

2. 

The  technology  allowed  for  easy 
review. 

82% 

89% 

86% 

67% 

3. 

I  was  able  to  access  the  training  with 
minimum  assistance. 

88% 

100% 

86% 

67% 

4. 

I  was  able  to  successfully  operate  the 
functionality  within  the  training. 

74% 

67% 

100% 

67% 

5. 

I  am  satisfied  with  the  technology 
interface. 

68% 

72% 

71% 

56% 

Average  Endorsement 

79% 

83% 

89% 

62% 

During  observations  and  post-training  debriefs  for  the  collaborative  scenario,  some 
usability  issues  were  uncovered  that  were  likely  explanations  for  the  lower  ratings  on  technology 
for  this  modality.  Participants  identified  interactions  with  the  virtual  radio  that  were  too 
cumbersome,  unintuitive,  or  different  from  interacting  with  the  radio  physically  in  the  real  world. 
After  the  beta  test,  all  usability  issues  identified  were  systematically  reviewed. 

Instructional  Design 

In  terms  of  instructional  design,  participants  were  asked  about  the  training  content  (e.g., 
the  accuracy  and  effectiveness  of  the  training  material)  and  the  presentation  (e.g.,  design, 
sequence,  and  motivational  effectiveness)  of  the  material.  As  shown  in  Table  11,  over  80%  of 
participants  endorsed  the  course  design  for  the  mobile  and  collaborative  scenario  modalities. 
Attitudes  were  less  positive  on  the  virtual  classroom  modality,  with  approximately  70%  of 
participants  indicating  favorability  of  the  course  design  for  this  modality.  The  majority  of 
participants  felt  that  the  virtual  classroom  course  design  did  not  motivate  them  to  learn,  and  only 
about  40%  of  the  participants  agreed  that  the  material  was  accurate  and  current. 
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Table  11 


Percentages  of  Favorable  Ratings  on  Instructional  Design  by  Training  Modality 


Overall 

Mobile 

Virtual 

Classroom 

Collaborative 

Scenario 

Instructional  Design 

(N  =  34) 

(N  =  18) 

(N  =  7) 

(N  =  9) 

1. 

The  session  objectives  were  met. 

91% 

94% 

86% 

89% 

2. 

The  material  in  this  lesson  was 
accurate  and  current. 

71% 

78% 

43% 

78% 

3. 

The  design  of  the  training  was  an 
effective  way  to  present  the  subject 
matter. 

74% 

67% 

71% 

89% 

4. 

The  material  was  presented  in  a  logical 
sequence  so  that  it  has  helped  me 
understand  the  subject  matter. 

91% 

100% 

86% 

78% 

5. 

The  media  (i.e.,  graphics  and  animated 
sequences)  appropriately  illustrate  the 
points  being  discussed. 

94% 

89% 

100% 

100% 

6. 

The  design  and  presentation  of 
material  motivated  me  to  learn. 

65% 

72% 

43% 

67% 

7. 

Overall,  I  am  pleased  with  the  way 
training  was  presented. 

74% 

72% 

71% 

78% 

Average  Endorsement 

80% 

82% 

71% 

83% 

During  post-training  debriefs,  participants  raised  concerns  about  some  of  the  virtual 
classroom  content  containing  unnecessary  details.  Those  participants  who  completed  both  the 
mobile  training  and  the  virtual  classroom  noted  that  this  was  especially  true  for  topics  that  were 
basic  and  had  already  been  covered  in  the  mobile  training.  As  a  result  the  length  of  the  virtual 
classroom  modality  was  too  long,  which  could  lead  to  learner  fatigue  and  loss  of  attention.  For 
the  low  ratings  on  the  item  about  the  material  being  accurate  and  current,  participants  were  likely 
referring  to  the  relevance  but  not  the  accuracy  of  the  content  because  the  training  content  is 
consistent  across  the  three  modalities.  In  light  of  this  finding,  the  structure  of  the  virtual 
classroom  was  revised  after  the  beta  test  to  increase  flexibility  for  the  instructor  to  adjust  the 
level  of  detail  in  presenting  the  material. 

On  the  collaborative  scenario,  it  should  also  be  noted  that,  despite  less  favorable 
perceptions  of  technology,  all  participants  favored  using  graphics  and  animated  sequences.  This 
result  further  indicates  that  any  technology-related  issues  in  the  collaborative  scenario  were 
likely  associated  with  user  functionality  rather  than  the  media. 

Perceived  Learning  and  Utility 

When  asked  about  learning  as  a  result  of  these  training  modalities  (Table  12),  the 
majority  of  participants  in  the  mobile  modality,  slightly  over  half  of  participants  in  the 
collaborative  scenario  modality  and  less  than  half  of  participants  in  the  virtual  classroom 
modality  indicated  they  “learned  a  lot”  from  the  training.  However,  across  all  modalities,  less 
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than  half  of  the  participants  reported  that  their  level  of  knowledge  and  perceived  ability  to  apply 
the  strategies  and  techniques  presented  to  an  actual  situation  changed  as  a  result  of  the  training. 


Table  12 


Percentages  of  Favorable  Ratings  on  Perceived  Learning  Outcomes  by  Training  Modality 


Learning 

Overall 
(N  =  34) 

Mobile 
(N  =  18) 

Virtual 
Classroom 
(N  =  7) 

Collaborative 
Scenario 
(N  =  9) 

1. 

Overall,  I  have  learned  a  lot  from  this 
training. 

63% 

75% 

43% 

56% 

2. 

My  knowledge  of  the  subject  AFTER 
taking  this  lesson  compared  to  PRIOR 
to  taking  this  lesson. 

41% 

50% 

29% 

33% 

3. 

My  ability  to  apply  the  strategies  and 
techniques  presented  to  an  actual 
situation  in  this  subject  AFTER  taking 
this  lesson  compared  to  PRIOR  to 
taking  this  lesson. 

25% 

22% 

14% 

33% 

Average  Endorsement 

43% 

49% 

29% 

41% 

Additionally,  slightly  more  than  half  of  the  participants  endorsed  the  practical  value  of 
the  training  and  its  application  on  the  job  across  all  modalities  (see  Table  13).  The  reported  lack 
of  learning,  training  utility,  and  learning  transfer  as  a  result  of  the  training  was  likely  due  to  the 
sample  used  for  the  beta  test.  Almost  all  participants  in  this  sample  were  25C  or  25U  Soldiers 
with  extensive  experience  with  the  JEM.  Thus,  the  low  levels  of  changes  in  knowledge  and 
ability,  perceived  utility,  and  learning  transfer  are  likely  the  result  of  a  pre-training  ceiling  effect, 
as  these  Soldiers  were  experienced  with  the  JEM  prior  to  engagement  in  the  training. 
Nevertheless,  participants  emphasized  during  debriefs  that  this  training  would  be  valuable  as 
refresher  training  or  an  introductory  training  for  lower-level  Soldiers. 
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Table  13 


Percentages  of  Favorable  Ratings  on  Perceived  Value  and  Utility  by  Training  Modality 


Overall 

Mobile 

Virtual 

Classroom 

Collaborative 

Scenario 

Value  /  Utility 

(N  =  34) 

(N  =  18) 

(N  =  7) 

(N  =  9) 

1. 

It  is  clear  to  me  that  the  people 
conducting  the  training  understand 
how  I  will  use  what  I  learn. 

71% 

83.3% 

71% 

44% 

2. 

This  training  was  relevant  to  my  job  in 
the  Army. 

79% 

89% 

71% 

67% 

3. 

I  believe  the  training  will  help  me  do 
my  current  job  in  the  Army  better. 

56% 

31% 

43% 

56% 

4. 

I  learned  something  I  can  apply 
immediately  to  my  work  in  the  Army. 

42% 

39% 

43% 

44% 

5. 

I  plan  to  use  what  I  learned  on  my  job 
in  the  Army. 

53% 

56% 

43% 

56% 

6. 

I  get  excited  when  I  think  about  trying 
to  use  my  new  learning  on  my  job  in 
the  Army. 

56% 

61% 

43% 

56% 

7. 

The  training  was  of  practical  value  to 
me. 

65% 

83% 

29% 

56% 

Average  Endorsement 

60% 

63% 

49% 

54% 

Enjoyment  and  Overall  Satisfaction 

Overall,  results  showed  that  the  training  was  perceived  positively  despite  a  few  targeted 
issues  mentioned  above.  As  shown  in  Table  14,  the  majority  of  participants  indicated  that  they 
enjoyed  the  training  and  would  recommend  it  to  others  across  all  three  training  modalities. 
Although  participants  in  the  virtual  classroom  reported  lower  levels  of  enjoyment  and  feelings  of 
“time  well  spent”  compared  to  the  other  two  modalities,  which  was  most  likely  due  to  concerns 
about  the  content  and  length  of  the  training,  they  would  still  recommend  the  training  program  to 
other  Soldiers. 

Table  14 


Percentages  of  Favorable  Ratings  on  Enjoyment  and  Overall  Satisfaction  by  Training 
Modality _ 


Enjoyment 

Overall 
(N  =  34) 

Mobile 
(N  =  18) 

Virtual 
Classroom 
(N  =  7) 

Collaborative 
Scenario 
(N  =  9) 

1. 

I  enjoyed  this  training  program. 

74% 

78% 

57% 

78% 

2. 

My  time  on  the  training  was  well 
spent. 

71% 

78% 

43% 

78% 

3. 

I  would  recommend  this  training 
program  to  other  Soldiers. 

77% 

83% 

71% 

67% 

Average  Endorsement 

74% 

80% 

57% 

74% 
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Discussion  and  Lessons  Learned 


The  goal  of  this  project  was  to  develop  prototype  training  leveraging  mobile,  virtual 
classroom  and  gaming-technology  that  could  be  used  as  a  test-bed  for  conducting  research  on 
training  and  assessment  strategies  outlined  in  the  ALM.  The  design  of  the  prototypes  aligned 
with  principles  of  the  learning  sciences  and  research  based  practices  for  assessment;  learning 
opportunities  were  situated  into  realistic  contexts;  assessments  were  administered  on  the  same 
platforms  as  were  used  to  deliver  training;  frequent  testing  during  training  was  employed;  and 
the  assessments  and  assessment  items  were  carefully  mapped  to  learning  objectives  to  ensure 
that  assessments  measured  the  constructs  of  interest. 

The  following  section  discusses  lessons  learned  from  developing  the  prototypes.  These 
lessons  feel  into  two  broad  categories:  (1)  transitioning  existing  training  content  into  a  virtual 
format,  and  (2)  using  assessments  in  training. 

Transitioning  Content  to  a  Virtual  Format 

The  first  set  of  lessons  learned  reflect  issues  pertaining  to  the  transfer  of  information  from 
the  classroom  version  of  the  training  to  the  virtual  version. 

A  significant  challenge  in  developing  the  training  content  was  gathering  sufficient 
information  regarding  how  operators  use  the  JEM  radio  in  the  field  and  common  troubleshooting 
tasks  that  are  likely  to  happen  when  operating  the  radio.  Although  the  design  team  had  access  to 
the  JEM  user  manual,  training  slides,  training  support  packages  and  an  ALC  instructor,  it  would 
have  been  beneficial  to  work  with  a  consistent,  core  group  of  SMEs  as  the  content  was 
developed.  This  would  have  allowed  the  research  team  to  draw  on  a  breadth  of  perspectives  and 
experiences  on  how  the  radio  is  used  in  the  field,  while  ensuring  consistency  in  the  training. 
While  there  was  access  to  one  individual  SME  throughout  the  process,  other  SMEs  rotated  in  and 
out.  For  certain  kinds  of  tasks,  for  example,  those  that  require  a  high  degree  of  judgment,  this 
breadth  in  perspective  was  beneficial.  Basic  instruction  in  the  use  of  a  piece  of  equipment, 
however,  requires  depth  of  expertise  and  consistency,  rather  than  breadth  of  perspective.  In 
hindsight,  it  may  have  been  more  effective  to  have  one  small,  core  group  of  JEM  experts  work 
closely  with  the  team  as  the  content  was  developed.  A  related  challenge  was  limited  access  to 
equipment,  specifically  the  AN/PRC  148  JEM  radio. 

Lesson  1:  Having  a  small  core  group  of  SMEs  throughout  the  project  is  a 

desirable  approach  for  training  content  development. 

One  major  consideration  during  training  development  was  determining  which  training 
content  would  go  into  each  modality.  Our  three  terminal  objectives  aligned  reasonably  well  with 
three  different  training  delivery  methods  due  to  the  features  of  these  training  methods  and  the 
characteristics  necessitated  by  the  objectives.  For  example,  the  ability  to  access  content  anytime 
and  anywhere  aligned  with  describing  basic  features,  components,  and  capabilities.  As  learners 
progressed  to  demonstrate  procedures  and  engage  in  troubleshooting,  the  interactive  features  of 
the  virtual  classroom  became  important.  The  development  of  the  ISD  map  was  an  important  step 
in  the  alignment  of  the  training  content  with  the  training  technologies. 
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Lesson  2:  The  learning  objectives  are  an  important  driver  of  modality  selection. 

A  final  challenge  was  trying  to  create  a  collaborative,  scenario-based  exercise  for  a  task 
that  is  not  inherently  collaborative.  This  issue  was  addressed  by  creating  a  concept  that  involved 
the  radio  being  passed  from  one  learner  to  another  and  encouraging  the  other  learners  to  provide 
input  into  the  task  via  a  live  chat  function.  This  concept  of  collaborating  with  peers  to  learn  the 
radio  met  the  high  level  learning  objective  outlined  in  the  ISD  Map.  In  hindsight,  a  task  that  is 
more  fundamentally  collaborative  in  nature  would  have  allowed  us  to  make  the  collaborative 
scenario  more  “true  to  life.” 

Lesson  3:  Collaborative  virtual  scenarios  may  be  more  effective  (“true-to-life”)  if 
the  actual  tasks  are  inherently  collaborative  in  nature. 

Using  Assessments  in  Training 

Another  category  of  lessons  learned  reflects  our  experiences  incorporating  an  array  of 
assessments  into  emerging  training  technologies.  First,  the  simulated  radio  that  was  built  for  this 
training  suite  afforded  the  opportunity  to  incorporate  more  interactive  assessment  items  than  a 
typical  multiple-choice  format.  One  challenge  in  implementing  this  was  ensuring  that  learners 
were  measured  on  their  skill  and  knowledge  of  the  radio,  not  in  using  the  training  /  assessment 
interface.  For  example,  if  the  task  required  the  learner  to  connect  a  cable  in  order  to  get  a  point, 
the  assessment  must  be  designed  in  such  a  way  that  learners  do  not  lose  points  simply  because 
they  did  not  know  how  to  use  the  interface  to  connect  the  cable  (i.e.,  manipulating  a  mouse  while 
entering  a  sequence  of  keys  on  the  keyboard).  Conversely,  assessment  developers  must  avoid 
providing  too  many  hints  or  instruction  about  how  to  perform  a  certain  function  because  they  can 
“give  away”  the  correct  answer.  Achieving  this  balance  requires:  (a)  a  skilled,  experienced 
assessment  developer  to  work  closely  with  technology  developers  and  (b)  more  usability  and 
pilot  testing  than  more  traditional  tests. 

Lesson  4:  Embedding  step-based  assessments  within  a  scenario-based  exercise 
requires  a  close  collaboration  between  assessment  and  technology  developers, 
and  more  usability  and  pilot  testing  than  more  traditional  test. 

Next,  one  of  the  prototype  assessments  developed  in  this  effort  was  a  CAT.  One  reason 
for  incorporating  a  CAT  into  the  training  context  was  to  determine  if  it  could  be  a  more  efficient 
way  to  measure  learning.  One  challenge  was  that  the  length  of  the  test  depends  on  the 
dimensionality  of  the  construct  because  a  separate  CAT  must  be  created  for  each  dimension.  For 
example,  if  a  learner  incorrectly  answers  a  difficult  item  on  dimension  X,  the  next  (easier)  item 
should  be  from  the  same  dimension  (or  construct)  -  not  from  a  completely  distinct  construct.  If 
training  content  is  multi-dimensional,  a  CAT  will  not  necessarily  be  more  efficient  than  a 
traditional  test.  A  related  challenge  was  that  because  there  was  no  way  to  know  the  true 
dimensionality  of  the  JEM  training  content,  assumptions  were  made  about  it  when  constructing 
the  CAT.  An  assumption  was  made  that  the  various  content  areas  are  related  to  each  other  and 
all  pertain  to  a  higher-order  general  ability  to  use  the  JEM.  The  different  content  areas  are 
represented  in  the  item  pool  and  the  same  number  of  items  from  each  content  area  is  drawn  for 
each  test.  However,  the  adaptive  mechanism  functions  across  content  areas  (i.e.,  the  item 
selection  from  one  content  area  may  be  based  on  the  response  to  an  item  from  a  different  content 
area).  It  should  be  noted  that  (a)  item  selection  is,  in  part,  based  on  item  difficulty  and  (b)  each 
content  area  contained  items  of  similar  levels  of  difficulty.  Thus,  because  it  was  assumed  that  all 
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content  areas  pertain  to  a  higher  order  general  ability  to  use  the  JEM,  item  selection  from  one 
content  area  (including  difficulty)  based  on  the  response  to  an  item  from  another  content  area 
should  not  be  problematic.  The  purpose  behind  governing  the  content  area  representation  was 
simply  to  counterbalance  the  items  across  content  areas.  However,  if  factor  loadings 
demonstrated  that  our  content  areas  did  not  load  onto  a  single  dimension,  the  results  would 
indicate  that  cross  content  item  selection  may  not  be  appropriate.  Specifically,  it  would  lead  to 
CATs  that  assess  trainees  unequally. 

Lesson  5:  Ideally,  assessment  items  should  be  pilot  tested  in  order  to  confirm  the 
dimensionality  prior  to  finalizing  the  CAT. 

Conclusions 

The  purpose  of  this  effort  was  to  develop  prototype  training  with  integrated  assessments 
that  aligned  with  the  goals  of  the  ALM.  For  this  particular  effort,  the  focus  was  on  redesigning 
the  current  training  programming  for  the  JEM  radio  to  better  meet  the  vision  of  the  ALM.  After 
creating  an  ISD  map  based  on  current  training  materials  and  SME  input,  our  project  team  was 
able  to  match  training  objectives  to  training  delivery  methods  (i.e.,  mobile,  virtual  classroom, 
and  collaborative  scenario),  based  on  the  alignment  of  the  objectives  and  the  affordances  of  the 
delivery  method.  Using  best  practices  from  the  field,  our  project  team  then  developed  the 
training  environments,  training  content,  and  respective  assessments.  Next,  we  sought  to  assess 
our  prototype  with  an  Army  audience.  The  prototype  training  and  assessments  were  beta-tested 
with  a  sample  of  SMEs  and  instructors  from  the  Signal  NCO  Academy  to  assess  reactions  to  the 
prototype  training.  Generally  speaking,  the  results  demonstrated  that  the  participants  expressed  a 
positive  view  of  the  training  technology,  instructional  design,  and  overall  enjoyment.  Although 
participants  did  not  express  favorable  ratings  for  learning  and  learning  outcomes,  this  is  likely 
due  to  (a)  the  simplistic  content  of  the  training  itself  (i.e.,  the  JEM  training  is  straight  forward) 
and  (b)  the  fact  that  many  participants  had  extensive  experience  with  the  JEM.  Because  this  data 
was  collected  during  a  beta  test,  these  results  should  be  considered  preliminary. 

The  prototypes  developed  for  this  effort  are  intended  to  be  used  in  future  research  on 
training  assessments  and  training  effectiveness  using  emerging  training  technologies. 
Specifically,  future  research  will  include  the  following  questions: 

•  Is  the  prototype  training  at  least  as  effective  as  the  traditional  classroom  training? 

•  Does  an  adaptive  assessment  provide  value  over  non-adaptive  assessment  in  a  training 
context? 

•  Are  interim  assessments  effective  for  learning? 

•  What  is  the  most  effective  “schedule”  of  assessment? 

•  Does  better  performance  in  the  simulation  translate  to  better  performance  with  a  real 
radio? 

•  Are  there  any  negative  effects  of  practicing  on  a  simulated  radio? 

Answers  to  these  questions  can  inform  implementation  of  ALM  in  that  they  will  increase 
our  understanding  of  the  most  appropriate  uses  of  the  training  and  assessment  technology 
resources  that  exist.  The  prototypes  described  here  will  allow  this  research  to  be  conducted  with 
tools  that  leverage  best  practices  in  training  assessments,  enhancing  the  quality  of  the  research. 
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APPENDIX  A 

INSTRUCTIONAL  DESIGN  MAP 


A-l 


TRAIN  II  -  ASSESSMENTS  PROJECT  INSTRUCTIONAL  DESIGN  MAP 

Learning  Objective  Decomposition  for  AN/PRC-148  JEM  T raining 


Main  Instructional  Goal  1.0 

Learner  will  demonstrate  the  proper  use  of  the  JEM  with  100%  accuracy  and  within 
rapid  timeframes  in  both  usual  and  unusual  conditions  and  under  duress  and 
constraints,  while  troubleshooting  issues  as  they  arise. 


Terminal  Objective  1.1 

Learner  will  describe  the  basic  features,  technical 
components  arid  capabilities  of  the  JEM,  identify 
and  explain  the  purpose  of  each  component, 
identify  and  explain  how  to  connect  external 
components,  and  recognize  problems  that  require 
general  troubleshooting. 


Mobile  Device 


Terminal  Objective  1.2 

Learner  will  demonstrate  accurate 
and  timely  operation  of  the  JEM  in 
usual  conditions,  including 
troubleshooting  as  required  in 
specific  circumstances. 


Terminal  Objective  1.3 

Learner  will  demonstrate  operation  of  the  JEM, 
including  troubleshooting  as  required,  and 
evaluate  other  team  members'  use  of  the  JEM 
through  guidance,  collaboration  and  after  action 
review,  in  usual  and  unusual  conditions. 


Virtual 

Classroom 


r  \ 

Collaborative 

Scenario 

V _ / 


Key 


^  Viaw^ 


Terminal  Objective  =  Delivery  Method 
Subordinate  Objectives  =  Content  Areas 
Enabling  Objective  =  Unit  of  Content 

Sub  Enabling  Objective  =  Smallest  Unit  of  Content 

Objective  is  related  to  troubleshooting 

Indicates  a  specified  view  of  the  JEM 

Objective  has  been  removed 


Objective  was  combined  with  another 
objective  in  the  Virtual  Classroom 


Objectives  were  combined  or  linked  in 
someway  in  the  storyboard  development 
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Subordinate  Objective  1.1.1 

Learner  will  define  relevantterminology,  explain  radio  safety  considerations,  and  describe  features, 
capabilities,  and  specifications  of  the  JEM. 
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Subordinate  Objective  1.1.2 

Learner  wil  I  identify  the  JEM's  components  {including  external  components),  explain  how  they 
are  properly  assembled  and  maintained,  describe  how  they  arc  used  in  the  context  of  the 
radio's  functionality,  and  state  any  relevant  information  such  as  reminders  or  troubleshooting. 


Enabling  Objective. 
B.l.1.2.1 


Identify  the  On/Off  button,  Volume  Control, 
and  Panic  Zeroize,  describe  their  use  in 
context,  and  state  any  relevant  information 
such  as  reminders  or  troubleshooting 


Identify  the  Liquid  Crystal  Display  (LCD), 
describe  its  use  in  context,  and  state  any 
relevant  information  such  as  reminders  or 
troubleshooting 


Bottom 

Vi«w  Enabling  Objective. 

C.  1.1. 2.1  (form.  Cl. 1.2.3) 

Identify  the  Battery,  describe  the  O-ring,  explain 
how  to  recognize  a  damaged  O  ring  and  state 
any  relevant  information 
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Subordinate  Objective  1.1.3 

Learner  will  explain  and  demonstrate  at  a  high  level  how  to  configure  and  operate  the  JEM  {including 
KeyFill)  as  well  as  how  to  access  each  display  screen  and  explain  the  significance  of  each. 
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A 

Obvious 
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USE  IN  CONJUNCTION  WITH 
CROSSWALK  DOCUMENTATION 


Terminal  Objective  1.2 

Learner  will  demonstrate  accurate  and  timely  operation 
of  the  JEM  in  usual  conditions,  including  troubleshooting 
as  required  in  specific  circumstances. 


Subordinate  0 

Learner  will  demons 
Up  procedures,  whilt 
required,  and  will  er 
and  peer-to-pe< 
appro 

bjective  1.2.1. 
tratc  the  JEM's  Start 
trouble  shooting  as 

»r  instruction  as 
priate. 

i 

Subordinate  Objective  1.2.2. 

Learner  will  demonstrate  how  to 
program  the  JEM's  Global  Parameters, 
while  trouble  shooting  as  required,  and 
will  engage  in  story-telling  and  peer-to- 
peer  instruction  as  appropriate. 

’ 

r 

Subordinate  Objective  1.2.3. 

Learner  will  demonstrate  the  JEM's  Key 
Management  procedures,  while  trouble 
shooting  as  required,  and  will  engage  in 
story-telling  and  peer-to-peer  instruction 
as  appropriate. 

Subordinate  Objective  1.2.4. 

Learner  will  demonstrate  how  to 
program  channels  on  the  JEM  ,  while 
trouble-shooting  as  required,  and  will 
engage  in  story-telling  and  peer-to-peer 
instruction  as  appropriate. 

1 

r 

Subordinate  Objective  1.2.5. 

Learner  will  demonstrate  how  to  set  up 
Group  Programming ,  while  trouble¬ 
shooting  as  required,  and  will  engage  in 
story-telling  and  peer-to-peer  instruction 
as  appropriate. 

! 

Subordinate  0 

Learner  will  demons 

emergency  progran 

arid  how  to  view  stat 
trouble-shooting  a 
engage  in  story-telli 
instruction  as 

bjective  1.2.6. 

rate  how  to  perform 
nming  with  the  JEM 
us  information,  while 
required,  and  will 
lg  and  peer-to-peer 
appropriate. 

A 


In  VC,  moved  to 
1.2.7 


-A 


Enabling  Objective. 

1.2.6 

Demonstrate  how  to  access  the 
screen  for  and  program  an 
Emergency  Beacon  and  share  stories 
about  real  emergencies  when  this  is 
used. 


Enabling  Objective. 

1.2.6.2 

Demonstrate  how  to  access  the 
screen  for  and  program  Situational 
Awareness. 


Enabling  Objective. 

1.2.6.3 

Demonstrate  how  to  access  the 
Status  screen  to  view  transmitted 


Objective  has  been  removed 

Subordinate  Objective  1.2.7. 
Learner  will  demonstrate  how  to  zeroize 
the  JEM  and  provide  a  context  for  a 
complete  zerioze  versus  a  channel 
zeroize  and  will  engage  in  story-telling 
and  peer-to-peer  interaction  as 
appropriate. 


Subordinate  Objective  1.2  8. 

Learner  will  demonstrate  how  to  clone 
JEM's  programming,  while  trouble 
shooting  as  required,  and  will  engage  in 
story  telling  and  peer  to  peer  interaction 
as  appropriate. 


Subordinate  Objective  1.2.9. 

Learner  will  demonstrate  the  JEM's  PC 
Toolkit  operating  procedures,  while 
trouble-shooting  as  required  and  will 
engage  in  story-telling  and  peer-to-peer 
instruction  as  appropriate. 


Subordinate  Objective  1.2.9 
(Form.  1.2.10) 

Learner  will  demonstrate  establishing 
communications  with  configured  radio 
using  the  PTT  with  another  peer,  several 
peers,  and  /or  with  the  virtual  instructor 


VC  Assessment 

Enabling  Objective. 

1.2. 9.1  (Form.  1.2.10.1) 

Through  deduction,  describe  the 
appropriate  corrective  action 
should  the  JEM  fail  to  establish 
communication 


SubEnabling  Objective. 

1.2.10.1.1 
Check  Power 
Check  Volume 
Check  Antenna  and  Antenna 
connection 

Check  Battery  Connection  (screws  /  O- 
ring) 

Check  Battery  Life 
Check  Internal  /  External  Mic 
Check  audio  /  keyfill  connector 
Check  all  programming  (BASIC, 
SINCGARS,  SINCGARS  FH  errors) 
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Subordinate  Objective  1.2.3. 

Learner  will  demonstrate  the  JEM's  Key  Management  procedures, 
while  trouble  shooting  as  required,  and  will  engage  in  story  telling 
and  peer-to-peer  instruction  as  appropriate. 
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Subordinate  Objective  1.2.4. 

Learner  will  demonstrate  how  to  program  channels  on  the  JCM  ,  while  trouble 
shooting  as  required,  and  will  engage  in  story-telling  and  peer-to-peer 
instruction  as  appropriate. 
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This  objective  has  been  added 

Objective  has  been  removed 

(taken  from  1.2.6,  which  was 
eliminated). 
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Subordinate  Objective  1.2  8. 

Learner  will  demonstrate  how  to  done  JEM's  programming,  while  trouble  shooting  as  required, 
and  will  engage  in  story-telling  and  peer-to-peer  interaction  as  appropriate. 


Enabling  Objective. 

1.2.8.1 

Share  stories  related  to  cloning, 
explaining  when  it  was  necessary 
and  how  it  facilitated  the  situation. 


Enabling  Objective. 

1.2.8. 2 

Demonstrate  obtaining  the  proper 
equipment  to  clone  two  radios  and 
set  up  the  radios  for  SEND  and 
RECEIVE 


Enabling  Objective. 

1.2.8.3 

Demonstrate  powering  on  both 
radios  arid  properly  configuring  the 
CLONC  TX  and  CLONE  RX  on  radios 
using  the  MODE  screen. 


i 


Enabling  Objective. 

1. 2.8.4 

Demonstrate  setting  both  radios  for 
internal  audio  and  enabling  the  Side 
Connector 


In  VC,  this  is 
combined  with 
1 .2.8.3 


A 


Enabling  Objective. 

1.2.8.5 

Demonstrate  connection  of  the 
cloning  cable  to  the  SEND  and 
RECEIVE  radios  using  the  proper  end 
of  the  cloning  cable  i 


In  VC,  this  is 
combined  with 
1. 2.8.3 


A 


Enabling  Objective. 

1.2.8.6 

Identify  the  screen  display  on  both 
radios  that  indicates  each  radio  is 
ready  for  cloning  arid  execute  the 
proper  step  to  begin  cloning. 


A 


In  VC,  this  is 
combined  with 
1. 2.8.6 


A 


A 


In  VC,  this  is 
combined  with 
1 .2.8.6 


A 


m 


in  VC,  this  is 
combined  with 
1. 2.8.1 


A 


Enabling  Objective. 

1.2.8.7 

Identify  the  screen  display  on  both 
radios  that  indicates  that  the  radios 
are  cloning. . 


Enabling  Objective. 

1.2.8. 9 

Demonstrate  the  proper  steps  to 
disconnect  the  cable  from  the 
RECEIVING  radio  in  the  correct 
timeframe 


Enabling  Objective. 

1.2.3.11 

Explain  which  information  will  not 
be  transferred  to  the  RECEIVE  radio 
and  would  need  to  be  added. 


Enabling  Objective. 

1. 2.8.8 

Identify  the  screen  display  that 
indicates  the  cloning  was  not 
successful  and  demonstrate  the 
appropriate  corrective  action. 


Enabling  Objective. 

1.2.8.10 

Explain  what  happens  if  the 
RECEIVE  radio  is  not 
disconnected  timely  and 
powered  on. 


j 


L 


In  VC,  this  is 
combined  with 
1. 2.8.6 


A 


Subordinate  Enabling  Objective. 

1.2.8.11.1 

Load  COMSEC  keys  to  RECEIVE  Radio 


Subordinate  Enabling  Objective. 

1.2.8.11.2 

Enter  Julian  Date  and  Zulu  time  in 
RECEIVE  Radio  for  SINCGARS  FH 
Channel  Programming 
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APPENDIX  B 

COLLABORATIVE  SCENARIO  -  VIGNETTE  SCRIPT 


B-l 


TRAIN  II  Assessments 
Collaborative  Scenario  -  Vignette  Script 

VIGNETTE  1  -  ALL  PLAYERS 

Starting  position  -  Top  view  of  all  Soldiers  indoors  at  the  FOB  preparing  for  a  mission.  Pan 
across  screen.  Unit  Commander  is  standing  in  front  of  the  room.  Seven  Soldiers  are  seated  in 
chairs  in  front  of  him  (Non  Player  Characters  (NPC):  Truck  Commander,  Driver,  Gunner, 
Convoy  Commander,  Players:  RTO,  two  Soldier  passengers).  Eight  radios  are  lined  up  on  a 
table  behind  the  Unit  Commander.  The  Unit  Commander  will  be  holding  a  Commo  card  and  a 
radio. 


Start  Action  1  -  Switch  to  side  scrolling  view.  Focus  on  Unit  Commander,  who  is  at  the 
front  of  the  room.  Unit  Commander  begins  to  talk  about  the  mission. 

© 

Sounds  -  Unit  Commander  says,  “Our  mission  today  is  a  PSD.  We  will  be  transporting 
the  Colonel  to  a  local  village  so  he  can  check  on  the  progress  of  a  new  school  under  construction, 
We  will  leave  in  approximately  2  hours.  As  you  see  behind  me,  all  the  radios  are  properly 
assembled  and  have  been  checked  for  any  hardware  problems.  Your  task  will  be  to  program  all 
the  channels  for  your  radio  based  on  the  information  on  the  Commo  card  and  then  clone  the  rest 
of  the  radios.” 

Continue  Action  1  -  Unit  commander  hands  Soldier  the  Commo  card  and  players  are 
seated  at  the  table  where  all  the  radios  are.  Image  fades. 

Background  image:  Black  and  white  faded  image  of  side  view  of  all  the  Soldiers  seated  in  the 
convoy  room.  At  the  bottom  of  the  screen,  image  of  three  players,  each  respective  player  is 
highlighted  with  “You”  just  below  the  Soldier  image. 


Enter  Text  -  Prepare  your  radio  for  the  mission  by  completing  all  the  necessary  start-up 
procedures.  Perform  comms  check.  Clone  a  radio.  You  have  30  minutes.  Text  fades. 
Vignette  task  list  remains  on  screen  in  upper  left  corner.  Current  task  is  highlighted, 
completed  tasks  are  checked  off.  Clock  is  displayed  at  bottom  right  corner,  ticking  down 
30  minutes. 


Begin  Radio  Interaction  -  In  center  view  of  the  screen,  the  radio  appears.  The  player 
now  has  control  of  radio  using  mouse  and  keypad.  Players  can  hear  each  other  via  headsets. 


Radio  Interaction:  The  player  is  able  to  utilize  the  virtual  keypad  to  manually  program  the  radio 
according  to  the  Commo  card.  Player  must  press  the  “submit”  button  when  programming  is 
complete.  This  is  the  correct  answer. 
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Radio  Interaction— Alternative :  If  length  of  time  needed  to  program  radio  fully  is  deemed  too 
long  for  this  vignette,  provide  players  with  partially  programmed  radios.  The  player  is  able  to 
use  the  virtual  keypad  to  manually  program  the  channels  that  are  yet  to  be  entered  according  to 
the  Commo  card.  Player  must  press  the  “submit”  button  when  programming  is  complete.  This 
is  the  correct  answer. 


Option  2  for  Radio  Interaction:  The  player  uses  the  virtual  keypad  to  manually  program  the 
radio,  but  makes  mistakes  below  a  certain  threshold.  Player  must  press  the  “submit”  button 


when  programming  is  complete.  This  is  an  incorrect  answer. 


Enter  Text  (for  correct  answer  only)  -  You  have  successfully  programmed  the  channels  for 
your  radio  and  performed  the  necessary  communications  checks.  Now  you  may  proceed  to 
the  cloning  task. 


Incorrect  Answer  /  Time  Runs  out:  Show  areas  for  incorrect  actions  performed. 


Start  Action  2  -  Scene  continues  in  the  room  where  the  radios  are  being  programmed. 
The  other  Soldiers  present  in  the  room  are  completing  various  radio  tasks.  Player  has  radio 
which  he  or  she  has  already  programmed,  second  radio,  and  cloning  cable  (may  include  other 
cables  and  accessories  in  the  list).  “Clone  a  radio”  now  highlighted  on  vignette  task  list. 

Radio  Interaction:  The  player  must  program  the  radios  first  for  cloning:  TX  and  Receive, 
followed  by  attaching  the  cloning  cable  correctly  and  press  PTT.  This  is  the  correct  answer. 


Incorrect  Answer  /  Time  Runs  out  /  Partial  Credit  Answer:  Display  animated  video  of 
cloning  the  radio. 
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TRAIN  II  Assessments 
Collaborative  Scenario  -  Vignette  Script 

VIGNETTE  2  -  MAIN  PLAYER  VIEW 

Starting  position  -  Top  view  of  six  passengers  mounted  in  a  vehicle  on  a  dirt  road.  Driver  NPC 
is  seated  in  the  Driver’s  Seat.  Colonel  NPC  is  seated  in  the  passenger  seat.  Gunner  is  in  position 
at  center  of  vehicle.  RTO  player  is  seated  in  middle  row  behind  Driver.  Soldier  player  1  is 
seated  in  middle  row  behind  Colonel.  Soldier  player  2  is  sitting  in  back  row  behind  Driver.  All 
players  are  equipped  with  radios  that  are  attached  with  an  external  MIC  (clipped  to  shoulder). 

Start  Action  1  -  Vehicle  moves  north  on  the  dirt  road.  Show  vehicle  coming  to  a  stop. 
Show  the  RTO  and  Soldier  players  dismounting  with  the  Colonel.  Image  fades. 

Background  image:  Black  and  white  faded  image  of  side  view  of  vehicle.  At  the  bottom  of  the 
screen,  Image  of  three  players,  the  main  player  (Player  1)  is  highlighted  with  “You”  just  below 
the  Soldier  image. 


Enter  Text  -  Your  convoy  needs  to  assist  another  Battalion  within  a  close  distance.  When  you 
are  about  a  half  kilometer  away,  no  one  is  able  to  communicate  through  the  radio. 


SHOW:  Soldier  reaching  for  the  radio,  attempts  to  make  call,  and  then  show  that  he  is  frustrated 
because  he  is  unable  to  communicate.  He  keeps  pushing  on  the  PTT  button  but  it  appears  to  be 
stuck. 


Enter  Knowledge  Check  -  What  is  the  most  likely  reason  why  no  one  is  able  to  communicate 
through  the  radio? 

A.  The  battery  is  not  properly  connected. 

B.  The  radios  are  set  to  different  CTCSS  tones. 

C.  The  side  connector  is  enabled. 

D.  Your  radio  has  a  “hot  mic”.  Correct  answer. 

Question  Fades. 

Enter  Knowledge  Check  -  Which  of  the  following  functions  could  have  prevented  the  “hot 
mic”  situation? 


A.  Squelch. 

B.  Transmit  (TX)  Timeout.  Correct  answer. 

C.  Clear  bypass  reception. 

D.  Repeater  delay. 
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Question  Fades. 

Enter  Text.  You  now  have  2  minutes  to  troubleshoot  the  situation.  Text  fades. 


& 


Begin  Radio  Interaction  1  -  In  center  view  of  the  screen,  the  radio  and  external  mic 
appear  fully  assembled.  The  main  player  now  has  control  of  radio  and  external  mic  using  mouse 
and  mainpad.  A  clock  is  displayed  at  the  bottom  right  corner  of  the  screen,  ticking  down  2 
minutes.  Players  can  hear  each  other  via  headsets. 


Option  1  for  Radio  Interaction:  The  player  is  able  to  utilize  the  virtual  mainpad  to  remove  the 
external  mic  and  then  switch  the  radio  to  internal  mic.  The  player  presses  the  MODE  key,  and 
then  presses  ENT  on  INT  AUDIO,  then  presses  ENT  again  to  confirm  the  selection.  The  player 
should  then  press  ESC  key  to  return  to  main  screen.  The  player  should  verify  that  the  headset 
icon  is  no  longer  on  the  screen.  The  player  must  press  the  “submit  ”  button  when  the  action  is 
complete.  This  is  the  correct  answer. 

Option  2  for  Radio  Interaction:  The  player  attempts  to  change  the  battery,  power  off  the  radio, 
reattach  the  antenna  or  change  the  programming  of  the  radio.  This  is  an  incorrect  answer. 


f1 


Incorrect  Answer  /  Time  Runs  out:  Display  animated  video  of  removing  the  external 
mic  and  switching  the  radio  to  internal  mic. 
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VIGNETTE  2  -  INACTIVE  PLAYER  VIEW 


Starting  position  -  Top  view  of  six  passengers  mounted  in  a  vehicle  on  a  dirt  road.  Driver  NPC 
is  seated  in  the  Driver’s  Seat.  Colonel  NPC  is  seated  in  the  passenger  seat.  Gunner  is  in  position 
at  center  of  vehicle.  RTO  player  is  seated  in  middle  row  behind  Driver.  Soldier  player  1  is 
seated  in  middle  row  behind  Colonel.  Soldier  player  2  is  sitting  in  back  row  behind  Driver.  All 
players  are  equipped  with  radios  that  are  attached  with  an  external  MIC  (clipped  to  shoulder). 

Start  Action  1  -  Vehicle  moves  north  on  the  dirt  road.  Show  vehicle  coming  to  a  stop. 
Show  the  RTO  and  Soldier  players  dismounting  with  the  Colonel.  Image  fades. 

Background  image:  Black  and  white  faded  image  of  side  view  of  vehicle.  At  the  bottom  of  the 
screen,  Image  of  three  players,  the  inactive  player  (Player  2  /  RTO  or  Player  3)  is  highlighted 
with  “You”  just  below  the  Soldier  image. 


Enter  Text  -  Your  convoy  needs  to  assist  another  Battalion  within  a  close  distance.  When  you 
are  about  a  half  kilometer  away,  no  one  is  able  to  communicate  through  the  radio. 


SHOW:  Soldier  reaching  for  the  radio,  attempts  to  make  call,  and  then  show  that  he  is  frustrated 
because  he  is  unable  to  communicate.  He  keeps  pushing  on  the  PTT  button  but  it  appears  to  be 
stuck. 


Enter  Knowledge  Check  -  What  is  the  most  likely  reason  why  no  one  is  able  to  communicate 
through  the  radio? 

A.  The  battery  is  not  properly  connected. 

B.  The  radios  are  set  to  different  CTCSS  tones. 

C.  The  side  connector  is  enabled. 

D.  Your  teammate’s  radio  has  a  “hot  mic”.  Correct  answer. 

Question  Fades. 

Enter  Knowledge  Check  -  Which  of  the  following  functions  could  have  prevented  the  “hot 
mic”  situation? 


A.  Squelch. 

B.  Transmit  (TX)  Timeout.  Correct  answer. 

C.  Clear  bypass  reception. 

D.  Repeater  delay. 

Question  Fades. 

Enter  Text.  The  main  player  now  has  2  minutes  to  troubleshoot  the  situation.  Text  fades. 
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(fj) 

^  The  Main  Player  Begins  Radio  Interaction  1  -  In  center  view  of  the  screen,  the  radio 
and  external  mic  appear  fully  assembled.  The  main  player  now  has  control  of  radio  and  external 
mic  using  mouse  and  mainpad.  A  clock  is  displayed  at  the  bottom  right  corner  of  the  screen, 
ticking  down  2  minutes.  Players  can  hear  each  other  via  headsets. 


Option  1  for  Radio  Interaction:  The  player  is  able  to  utilize  the  virtual  mainpad  to  remove  the 
external  mic  and  then  switch  the  radio  to  internal  mic.  The  player  presses  the  MODE  key,  and 
then  presses  ENT  on  INT  AUDIO,  then  presses  ENT  again  to  confirm  the  selection.  The  player 
should  then  press  ESC  key  to  return  to  main  screen.  The  player  should  verify  that  the  headset 
icon  is  no  longer  on  the  screen.  The  player  must  press  the  “submit”  button  when  the  action  is 
complete.  This  is  the  correct  answer. 

Option  2  for  Radio  Interaction:  The  player  attempts  to  change  the  battery,  power  off  the  radio, 
reattach  the  antenna  or  change  the  programming  of  the  radio.  This  is  an  incorrect  answer. 


NOTE:  For  the  inactive  players,  all  above  options  have  a  top  view  (as  if  looking  over  one’s 
shoulder). 


•> 


Incorrect  Answer  /  Time  Runs  out:  Display  animated  video  of  removing  the  external 
mic  and  switching  the  radio  to  internal  mic. 
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TRAIN  II  Assessments 
Collaborative  Scenario  -  Vignette  Script 

VIGNETTE  3  -  MAIN  PLAYER  VIEW 

Starting  position  -  (Picking  up  from  Vignette  2  —  dismounted  from  the  vehicle).  Three  Soldier 
passengers  are  dismounted  and  in  the  operational  environment.  Player  2  (RTO)  and  Player  3  are 
equipped  with  a  radio  that  is  attached  with  an  external  MIC  (clipped  to  shoulder).  Player  1  is 
equipped  with  a  radio  without  the  external  MIC. 


Start  Action  1  -  All  players  hear  through  their  radio  a  code  word:  Blue  Thunder. 
Sounds  -  Play  NPC  voice  saying  “Blue  Thunder”  through  the  radio. 


Background  image:  Black  and  white  faded  image  of  Soldier  player  2  (RTO)  frozen  in  the 
position  of  communication  attempt.  At  the  bottom  of  the  screen,  the  main  player  (Player  2  / 
RTO)  is  highlighted  with  “You”  just  below  the  soldier  image. 


Enter  Text  -  You  have  just  heard  a  code  word  sent  out  through  the  higher  headquarter 
indicating  there  has  been  a  COMSEC  compromise  and  everyone  will  now  have  to  change  the 
Julian  date  as  a  result.  Text  fades. 


Enter  Knowledge  Check  -  What’s  the  purpose  of  changing  the  Julian  date  in  the  case  of  a 
COMSEC  compromise? 

A.  To  avoid  the  interception  of  local  radio  frequencies. 

B.  To  force  the  FH  communications  to  be  unsynchronized  with  the  original  Julian  Date. 
Correct  answer. 

C.  To  adjust  the  length  of  time  before  the  transmission  automatically  ends. 

D.  To  force  a  pause  during  extended  communications  to  enable  all  the  radios  to 
synchronize  themselves.  Question  Fades. 

Enter  Text  -  Blue  Thunder  =  2456185.5.  Begin  working  with  the  radio  now  to  troubleshoot  the 
issue.  You  have  2  minutes.  Text  fades. 


& 


Begin  Radio  Interaction  -  In  center  view  of  the  screen,  the  radio  appears.  The  player 
now  has  control  of  radio  using  mouse  and  keypad.  The  player  must  change  the  Julian  date.  A 
clock  is  displayed  at  the  bottom  right  comer  of  the  screen,  ticking  down  2  minutes.  Players  can 
hear  each  other  via  headsets. 
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Option  1  for  Radio  Interaction:  The  player  changes  the  Julian  date  to  the  one  provided  in  the 
text  defining  the  code  word.  Player  must  start  at  the  Default  screen  and  must  press  ALT  & 
MODE  to  access  the  MAIN  MENU.  Player  must  press  ENT  to  access  the  Programming  screen, 
and  scroll  using  the  Up  arrow  to  outline  the  GLOBAL  screen.  When  this  option  is  outlined, 
player  must  press  ENT.  On  the  GLOBAL  screen  player  must  scroll  down  to  outline  SET 
CLOCK,  then  press  ENT.  Once  on  the  SET  CLOCK  screen,  the  DAY  option  will  be  outlined. 
Player  must  press  ENT  to  change  the  Julian  Day.  Use  the  Up  and  Down  arrows  to  scroll 
through  number  values  for  the  highlighted  digit.  To  change  the  second  digit,  player  must  press 
ALT  and  either  Up  or  Down  Arrow,  then  use  the  Up  or  Down  arrows  to  scroll  through  the  values 


(0-9).  When  the  correct  Julian  Date  is  set,  player  should  press  ENT.  This  is  the  correct  answer. 


Option  2  for  Radio  Interaction:  The  player  is  unable  to  access  the  programming  or  global 


screens,  etc.  This  is  an  incorrect  answer. 


Incorrect  Answer  /  Time  Runs  out:  Display  animated  video  of  correctly  changing  the 
Julian  date. 
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VIGNETTE  2  -  INACTIVE  PLAYER  VIEW 


Starting  position  -  (Picking  up  from  Vignette  2 — dismounted  from  the  vehicle).  Three  Soldier 
passengers  are  dismounted  and  in  the  operational  environment.  Player  2  (RTO)  and  Player  3  are 
equipped  with  a  radio  that  is  attached  with  an  external  MIC  (clipped  to  shoulder).  Player  1  is 
equipped  with  a  radio  without  the  external  MIC. 


Start  Action  1  -  All  players  hear  through  their  radio  a  code  word:  Blue  Thunder. 
Sounds  -  Play  NPC  voice  saying  “Blue  Thunder”  through  the  radio. 


Background  image:  Black  and  white  faded  image  of  Soldier  player  2  (RTO)  frozen  in  the 
position  of  communication  attempt.  At  the  bottom  of  the  screen,  the  inactive  player  (Player  1  or 
Player  3)  is  highlighted  with  “You”  just  below  the  soldier  image. 


Enter  Text  -  You  have  just  heard  a  code  word  sent  out  through  the  higher  headquarter 
indicating  there  has  been  a  COMSEC  compromise  and  everyone  will  now  have  to  change  the 
Julian  date  as  a  result.  Text  fades. 


Enter  Knowledge  Check  -  What’s  the  purpose  of  changing  the  Julian  date  in  the  case  of  a 
COMSEC  compromise? 

A.  To  avoid  the  interception  of  local  radio  frequencies. 

B.  To  force  the  FH  communications  to  be  unsynchronized  with  the  original  Julian  Date. 
Correct  answer. 

C.  To  adjust  the  length  of  time  before  the  transmission  automatically  ends. 

D.  To  force  a  pause  during  extended  communications  to  enable  all  the  radios  to 
synchronize  themselves.  Question  Fades. 

Enter  Text  -  Blue  Thunder  =  2456185.5.  The  RTO  will  now  have  2  minutes  to  work  with  the 
radio  to  troubleshoot  the  issue.  Text  fades. 

(fj 

Main  Player  Begins  Radio  Interaction  -  In  center  view  of  the  screen,  the  radio 
appears.  The  player  now  has  control  of  radio  using  mouse  and  keypad.  The  player  must  change 
the  Julian  date.  A  clock  is  displayed  at  the  bottom  right  comer  of  the  screen,  ticking  down  2 
minutes.  Players  can  hear  each  other  via  headsets. 

Option  1  for  Radio  Interaction:  The  player  changes  the  Julian  date  to  the  one  provided  in  the 
text  defining  the  code  word.  Player  must  start  at  the  Default  screen  and  must  press  ALT  & 
MODE  to  access  the  MAIN  MENU.  Player  must  press  ENT  to  access  the  Programming  screen, 


B-10 


and  scroll  using  the  Up  arrow  to  outline  the  GLOBAL  screen.  When  this  option  is  outlined, 
player  must  press  ENT.  On  the  GLOBAL  screen  player  must  scroll  down  to  outline  SET 
CLOCK,  then  press  ENT.  Once  on  the  SET  CLOCK  screen,  the  DAY  option  will  be  outlined. 
Player  must  press  ENT  to  change  the  Julian  Day.  Use  the  Up  and  Down  arrows  to  scroll 
through  number  values  for  the  highlighted  digit.  To  change  the  second  digit,  player  must  press 
ALT  and  either  Up  or  Down  Arrow,  then  use  the  Up  or  Down  arrows  to  scroll  through  the  values 


(0-9).  When  the  correct  Julian  Date  is  set,  player  should  press  ENT.  This  is  the  correct  answer. 


Option  2  for  Radio  Interaction:  The  player  is  unable  to  access  the  programming  or  global 
screens,  etc.  This  is  an  incorrect  answer. 


NOTE:  For  the  inactive  players,  all  above  options  have  a  top  view  (as  if  looking  over  one’s 
shoulder). 


«> 


Incorrect  Answer  /  Time  Runs  out:  Display  animated  video  of  correctly  changing  the 
Julian  date. 
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TRAIN  II  Assessments 
Collaborative  Scenario  -  Vignette  Script 

VIGNETTE  4  -  MAIN  PLAYER  VIEW 

Starting  position  -  (Same  as  starting  position  in  Vignette  3  —  dismounted  from  the  vehicle). 
All  three  players  are  dismounted  with  the  Colonel  (NPC)  in  the  operational  environment.  Player 
2  (RTO)  and  Player  3  are  equipped  with  a  radio  that  is  attached  with  an  external  MIC  (clipped  to 
shoulder).  Player  1  is  equipped  with  a  radio  without  the  external  MIC. 


Start  Action  1  -  Start  scene  with  side  scrolling  view  with  image  of  three  players  in  the 
desert  with  the  operational  environment  surrounding  them.  Included  in  the  operational 
environment  is  a  vehicle  about  Vi  km  away  (not  their  own  vehicle).  Zoom  to  a  full-body  image 
of  the  main  player  (first-person  view  -  the  player  is  holding  a  radio  looking  at  the  vehicle  in 
short  distance).  Off  in  the  distance  behind  the  Soldier,  an  explosion  happens  [include  visual 
display  of  explosion  as  well  as  an  audio] .  The  main  player  gets  startled  as  he  hears  the  explosion 
sound  and  then  drops  the  radio. 


Sounds  -  Play  a  loud  explosion. 


Continue  Action  1  -  Vehicle  debris  and  shrapnel  spreads  from  the  explosion. 
Immediately  after  the  explosion,  an  NPC  is  running  towards  the  main  player  in  the  distance 
shouting  for  help. 


Sounds  -  Play  another  NPC  saying  “Someone  call  MEDEVAC.” 


Continue  Action  1  -  The  main  player  (first-person  view)  picks  up  radio  and  looks  down 
at  it.  Zoom  in  to  show  that  the  battery  is  slightly  twisted  and  radio  is  powered  off.  He  reattaches 
the  battery  and  powers  on  the  radio.  Image  fades. 

Background  image:  Black  and  white  faded  image  of  in  the  dessert  with  vehicle  on  fire  in  the 
background.  At  the  bottom  of  the  screen,  image  of  three  players,  the  main  player  (Player  3)  is 
highlighted  with  “You”  just  below  the  Soldier  image. 


Enter  Text  -  As  you  can  see,  your  radio  is  powered  off  from  the  battery  becoming  loose. 
Answer  this  question.  Text  fades. 


Enter  Knowledge  Check  -  What  most  likely  happened  to  your  radio  during  the  explosion  when 
the  battery  was  disconnected? 


A.  COMSEC  keys  have  been  lost.  Correct  answer. 

B.  The  radio  overheated. 
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C.  The  Julian  date  was  entered  incorrectly. 

D.  Programming  has  been  lost. 

Question  Fades. 


Enter  Knowledge  Check  -  You  need  to  use  this  radio  to  call  MedEvac  immediately.  How 
should  you  proceed? 

A.  Run  back  to  the  truck  to  reload  COMSEC  keys. 

B.  Turn  to  the  preprogrammed  channel  with  the  MedEvac  frequency.  Correct  answer. 

C.  Program  the  radio  and  manually  enter  the  MedEvac  frequency  in  SINCGARS. 

D.  Activate  an  emergency  beacon. 

Question  Fades. 

Enter  Text  -  You  need  to  use  this  radio  to  call  MedEvac  immediately  and  have  1  minute  to 
complete  this  task.  Text  fades. 


^  Begin  Radio  Interaction  -  In  center  view  of  the  screen,  the  radio  appears.  The  main 
player  now  has  control  of  radio  using  mouse  and  mainpad.  A  clock  is  displayed  at  the  bottom 
right  corner  of  the  screen,  ticking  down  60  seconds.  Players  can  hear  each  other  via  headsets. 


Option  1  for  Radio  Interaction:  The  player  is  able  to  utilize  the  virtual  keypad  to  switch  to  the 
unsecure  pre-programmed  channel  with  the  MedEvac  frequency  and  press  PTT.  The  player 
must  select  this  option.  This  is  the  correct  answer. 


Option  2  for  Radio  Interaction:  The  player  is  also  able  to  manipulate  the  power  ON/OFF  button 
to  reboot  the  radio.  The  player  may  turn  off  the  power,  disconnect  the  battery,  reconnect  the 
battery,  and  then  power  the  radio  back  on.  This  is  an  incorrect  answer. 


Option  3  for  Radio  Interaction:  The  player  is  also  able  to  utilize  the  virtual  keypad  to  activate 
the  emergency  beacon.  The  player  may  press  MODE  to  open  the  menu.  Then,  he  may  highlight 
and  select  BEACON  to  activate  the  alarm.  This  is  an  incorrect  answer. 


Option  4  for  Radio  Interaction:  The  player  is  also  able  to  utilize  the  virtual  keypad  to  attempt  to 
reprogram  the  radio.  This  is  an  incorrect  answer. 


f1 


Incorrect  Answer  /  Time  Runs  out  /  Partial  Credit  Answer:  Display  animated  video  of 
switching  to  the  preprogrammed  channel  and  calling  MEDEVAC. 
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VIGNETTE  4  -  INACTIVE  PLAYER  VIEW 


Starting  position  -  (Same  as  starting  position  in  Vignette  3 — dismounted  from  the  vehicle).  All 
three  players  are  dismounted  with  the  Colonel  (NPC)  in  the  operational  environment.  They  are 
each  equipped  with  a  radio  that  is  attached  with  an  external  MIC  (clipped  to  shoulder). 


Start  Action  1  -  Start  scene  with  side  scrolling  view  with  image  of  three  players  in  the 
desert  with  the  operational  environment  surrounding  them.  Included  in  the  operational 
environment  is  a  vehicle  about  Vi  km  away  (not  their  own  vehicle).  Zoom  to  a  full-body  image 
of  the  main  player  (second-person  view).  Off  in  the  distance  behind  the  Soldier,  an  explosion 
happens  [include  visual  display  of  explosion  as  well  as  an  audio].  The  main  player  gets  startled 
as  he  hears  the  explosion  sound  and  then  drops  the  radio. 


Sounds  -  Play  a  loud  explosion. 


Continue  Action  1  -  Vehicle  debris  and  shrapnel  spreads  from  the  explosion. 
Immediately  after  the  explosion,  an  NPC  is  running  towards  the  main  player  in  the  distance 
shouting  for  help. 


Sounds  -  Play  another  NPC  saying  “Someone  call  MEDEVAC.” 


Continue  Action  1  -  The  main  player  (second-person  view)  picks  up  radio  and  looks 
down  at  it.  Zoom  in  to  show  that  the  battery  is  slightly  twisted  and  radio  is  powered  off.  He 
reattaches  the  battery  and  powers  on  the  radio.  Image  fades. 

Background  image:  Black  and  white  faded  image  of  in  the  dessert  with  vehicle  on  fire  in  the 
background.  At  the  bottom  of  the  screen,  image  of  three  players,  the  inactive  player  (Player  1  & 
Player  2  /  RTO)  is  highlighted  with  “You”  just  below  the  Soldier  image. 


Enter  Text  -  As  you  can  see,  your  teammate’s  radio  is  powered  off  from  the  battery  becoming 
loose.  Answer  this  question.  Text  fades. 

Enter  Knowledge  Check  -  What  most  likely  happened  to  your  teammate’s  radio  during  the 
explosion  when  the  battery  was  disconnected? 

E.  COMSEC  keys  have  been  lost.  Correct  answer. 

F.  The  radio  overheated. 

G.  The  Julian  date  was  entered  incorrectly. 

H.  Programming  has  been  lost. 

Question  Fades. 
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Enter  Knowledge  Check  -  Your  teammate  needs  to  use  this  radio  to  call  MedEvac 
immediately.  How  should  your  teammate  proceed? 

E.  Run  back  to  the  truck  to  reload  COMSEC  keys. 

F.  Turn  to  the  preprogrammed  channel  with  the  MedEvac  frequency.  Correct  answer. 

G.  Program  the  radio  and  manually  enter  the  MedEvac  frequency  in  SINCGARS. 

H.  Activate  an  emergency  beacon. 

Question  Fades. 

Enter  Text  -  The  main  player  needs  to  use  this  radio  to  call  MedEvac  immediately  and  has  1 
minute  to  complete  this  task.  Text  fades. 


Main  Player  Begins  Radio  Interaction  -  In  center  view  of  the  screen,  the  radio 
appears.  The  main  player  now  has  control  of  radio  using  mouse  and  mainpad.  A  clock  is 
displayed  at  the  bottom  right  comer  of  the  screen,  ticking  down  60  seconds.  Players  can  hear 
each  other  via  headsets. 


Option  1  for  Radio  Interaction:  The  player  is  able  to  utilize  the  virtual  keypad  to  switch  to  the 
unsecure  pre-programmed  channel  with  the  MedEvac  frequency  and  press  PTT.  The  player 
must  select  this  option.  This  is  the  correct  answer. 


Option  2  for  Radio  Interaction:  The  player  is  also  able  to  manipulate  the  power  ON /  OFF 
button  to  reboot  the  radio.  The  player  may  turn  off  the  power,  disconnect  the  battery,  reconnect 


the  battery,  and  then  power  the  radio  back  on.  This  is  an  incorrect  answer. 


Option  3  for  Radio  Interaction:  The  player  is  also  able  to  utilize  the  virtual  keypad  to  activate 
the  emergency  beacon.  The  player  may  press  MODE  to  open  the  menu.  Then,  he  may  highlight 


and  select  BEACON  to  activate  the  alarm.  This  is  an  incorrect  answer. 


Option  4  for  Radio  Interaction:  The  player  is  also  able  to  utilize  the  virtual  keypad  to  attempt  to 
reprogram  the  radio.  This  is  an  incorrect  answer. 


NOTE:  For  the  inactive  players,  all  above  options  have  a  top  view  (as  if  looking  over  one’s 
shoulder). 


q 
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Incorrect  Answer  /  Time  Runs  out  /  Partial  Credit  Answer:  Display  animated  video  of 
switching  to  the  preprogrammed  channel  and  calling  MEDEVAC. 
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TRAIN  II  Assessments 
Collaborative  Scenario  -  Vignette  Script 

VIGNETTE  5  -  MAIN  PLAYER  VIEW 

Starting  position  -  (Picking  up  from  Vignette  4 — dismounted  from  the  vehicle).  Three  players 
are  dismounted  and  in  the  operational  environment.  Each  player  is  equipped  with  a  radio  that  is 
attached  with  an  external  MIC  (clipped  to  shoulder). 


Start  Action  1  -  Stall  scene  with  side  scrolling  view  with  image  of  three  players  in  the 
desert  with  the  operational  environment  surrounding  them.  Soldiers  walk  headed  east  towards 
their  vehicle.  They  reenter  the  vehicle.  Switch  to  top  view. 

Position  (once  at  the  vehicle)  -  Top  view  of  six  passengers  mounted  in  a  vehicle  on  a  dirt  road. 
Driver  NPC  is  seated  in  the  Driver’s  Seat.  Colonel  NPC  is  seated  in  the  passenger  seat.  Gunner 
is  standing  on  the  platform  directly  behind  between  the  Driver  and  the  Colonel.  Player  2  (RTO) 
is  seated  in  middle  row  behind  Driver.  Soldier  player  1  is  seated  in  middle  row  behind  Colonel. 
Soldier  player  3  is  sitting  in  back  row  behind  Driver.  Each  of  the  players  is  equipped  with  a 
radio  that  is  attached  with  an  external  MIC  (clipped  to  shoulder). 


Continue  Action  1  -  Vehicle  moves  north  on  the  dirt  road  towards  the  village. 
Sounds  -  Play  Colonel  saying,  “You  should  do  the  key  fill  now.” 


Continue  Action  1  -  The  RTO  (first-person  view)  takes  the  SKL  device  out  of  his  toolkit. 
Image  fades. 

Background  image:  Black  and  white  faded  image  of  Soldier  passenger  looking  at  the  radio  and 
SKL  device.  At  the  bottom  of  the  screen,  image  of  three  players,  the  main  player  (Player  2  / 
RTO)  is  highlighted  with  “You”  just  below  the  soldier  image. 


Enter  Text.  Begin  working  with  the  radio  now  to  reload  the  COMSEC  keys.  You  have  10 
minutes.  Text  fades. 


6* 


Begin  Radio  Interaction  1  -  In  center  view  of  the  screen,  the  radio  appears  fully 
assembled.  The  main  player  now  has  control  of  radio  using  mouse  and  keypad.  A  clock  is 
displayed  at  the  bottom  right  comer  of  the  screen,  ticking  down  15  minutes.  Players  can  hear 
each  other  via  headsets. 
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Option  1  for  Radio  Interaction:  The  player  is  able  to  manipulate  the  radio  keypad.  The  player 
should  prepare  the  radio  for  keyfill  (access  Key  fill  Screen  by  pressing  ALT+MODE),  connect  the 
Fill  cable  from  the  SKL  device  to  the  radio  ( the  SKL  device  has  already  been  prepared  for  the 
player),  and  press  PTT  to  load  keys.  This  is  the  correct  answer. 

Option  2  for  Radio  Interaction:  The  player  does  not  perform  any  of  the  necessary  steps 
preparing  the  radio  for  COMSEC  loading.  This  is  an  incorrect  answer. 

Option  3  for  Radio  Interaction:  The  player  either  successfully  prepares  the  radio  OR 
successfully  connects  the  Fill  cable  between  the  two  devices.  This  is  a  partial  credit  answer. 


P 


Incorrect  Answer  /  Time  Runs  out  /  Partial  Credit:  Display  animated  video  of  correct 
procedure  for  preparing  and  loading  COMSEC  keys. 


Start  Action  2  -  Top  view  of  Soldier  passenger  looking  down  at  the  radio.  Zoom  to  radio 
LCD  to  display  keyfill  error.  Zoom  back  out  to  show  Soldier  passenger  examining  the  radio. 

Image  fades. 


Enter  Text  -  As  you  can  see,  there  was  an  error  when  attempting  to  reload  the  COMSEC  keys 
on  your  radio.  Answer  this  question.  Text  fades. 

Enter  Knowledge  Check  -  Select  the  most  likely  reason(s)  why  you  are  receiving  the  keyfill 
error  message? 


I.  The  o-ring  is  corroded.  Correct  answer. 

J.  The  cable  is  damaged.  Correct  answer. 

K.  The  battery  is  not  secure. 

L.  The  key  is  invalid. 

Question  Fades. 

Enter  Text.  Begin  working  with  the  radio  now  to  troubleshoot  reloading  the  COMSEC  keys. 
You  have  5  minutes.  Text  fades. 


O 


Begin  Radio  Interaction  2  -  In  center  view  of  the  screen,  the  radio  connected  to  the 
SKL  device  is  shown.  It’s  shown  that  there  is  an  error  message  on  the  LCD  screen.  The  main 
player  continues  to  have  control  of  radio  using  mouse  and  keypad.  A  clock  is  displayed  at  the 
bottom  right  comer  of  the  screen,  continuing  the  countdown  of  15  minutes  from  previous 
interaction.  Players  can  hear  each  other  via  headsets. 
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Option  1  for  Radio  Interaction:  The  player  is  able  to  disconnect  the  keyfill  cable  and  replace  it 
with  a  spare  cable  (from  his  toolkit).  Then  he  will  access  Keyfill  Screen  by  pressing 
ALT+MODE  and  press  PTT  to  load  keys.  This  is  the  correct  answer.  Show  radio  LCD  saying 
“Fill  Success.” 

Option  2  for  Radio  Interaction:  The  player  does  not  replace  the  keyfill  cable.  This  is  an 
incorrect  answer. 

m 

Incorrect  Answer  /  Time  Runs  out:  Display  text  “Correct  action:  Replace  the  damaged 
keyfill  cable.”  Text  fades. 
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VIGNETTE  5  -  INACTIVE  PLAYER  VIEW 


Starting  position  -  (Picking  up  from  Vignette  4  —  dismounted  from  the  vehicle).  Three  Soldier 
passengers  are  dismounted  and  in  the  operational  environment.  One  Soldier  is  equipped  with  a 
radio  that  is  attached  with  an  external  MIC  (clipped  to  shoulder). 


Start  Action  1  -  Start  scene  with  side  scrolling  view  with  image  of  three  players  in  the 
desert  with  the  operational  environment  surrounding  them.  Soldiers  walk  headed  east  towards 
their  vehicle.  They  reenter  the  vehicle.  Switch  to  top  view. 

Position  (once  at  the  vehicle)  -  Top  view  of  six  passengers  mounted  in  a  vehicle  on  a  dirt  road. 
Driver  NPC  is  seated  in  the  Driver’s  Seat.  Colonel  NPC  is  seated  in  the  passenger  seat.  Gunner 
is  standing  on  the  platform  directly  behind  between  the  Driver  and  the  Colonel.  Player  2  (RTO) 
is  seated  in  middle  row  behind  Driver.  Soldier  player  1  is  seated  in  middle  row  behind  Colonel. 
Soldier  player  3  is  sitting  in  back  row  behind  Driver.  Each  of  the  players  is  equipped  with  a 
radio  that  is  attached  with  an  external  MIC  (clipped  to  shoulder). 


Continue  Action  1  -  Vehicle  moves  north  on  the  dirt  road  towards  the  village. 
Sounds  -  Play  Colonel  saying,  “You  should  do  the  key  fill  now.” 


Continue  Action  1  -  The  RTO  (second-person  view)  takes  the  SKL  device  out  of  his 
toolkit.  Image  fades. 

Background  image:  Black  and  white  faded  image  of  Soldier  passenger  looking  at  the  radio  and 
SKL  device.  At  the  bottom  of  the  screen,  image  of  three  players,  the  inactive  player  (Player  1  or 
Player  3)  is  highlighted  with  “You”  just  below  the  soldier  image. 


Enter  Text.  The  main  player  now  has  60  seconds  to  reload  the  COMSEC  keys.  Text  fades. 
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Main  Player  Begins  Radio  Interaction  1  -  In  center  view  of  the  screen,  the  radio 
appears  fully  assembled.  The  main  player  now  has  control  of  radio  using  mouse  and  keypad.  A 
clock  is  displayed  at  the  bottom  right  comer  of  the  screen,  ticking  down  15  minutes.  Players  can 
hear  each  other  via  headsets. 


Option  1  for  Radio  Interaction:  The  player  is  able  to  manipulate  the  radio  keypad.  The  player 
should  prepare  the  radio  for  key  fill  (access  Key  fill  Screen  by  pressing  ALT+MODE),  connect  the 
Fill  cable  from  the  SKL  device  to  the  radio  (the  SKL  device  has  already  been  prepared  for  the 
player),  and  press  PTT  to  load  keys.  The  player  must  select  this  option.  This  is  the  correct 
answer. 
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Option  2  for  Radio  Interaction:  The  player  does  not  perform  any  of  the  necessary  steps 
preparing  the  radio  for  COMSEC  loading.  This  is  an  incorrect  answer. 

Option  3  for  Radio  Interaction:  The  player  either  successfully  prepares  the  radio  OR 
successfully  connects  the  Fill  cable  between  the  two  devices.  This  is  a  partial  credit  answer. 


NOTE:  For  the  inactive  players,  all  above  options  have  a  top  view  (as  if  looking  over  one’s 
shoulder). 


Incorrect  Answer  /  Time  Runs  out  /  Partial  Credit:  Display  animated  video  of  correct 
procedure  for  preparing  and  loading  COMSEC  keys. 


Start  Action  2  -  Top  view  of  Soldier  passenger  looking  down  at  the  radio.  Zoom  to  radio 
LCD  to  display  keyfill  error.  Zoom  back  out  to  show  Soldier  passenger  examining  the  radio. 

Image  fades. 


Enter  Text  -  As  you  can  see,  there  was  an  error  when  attempting  to  reload  the  COMSEC  keys 
on  your  radio.  Answer  this  question.  Text  fades. 

Enter  Knowledge  Check  -  Select  the  most  likely  reason(s)  why  your  teammate  is  receiving  the 
keyfill  error  message? 


A.  The  o-ring  is  corroded.  Correct  answer. 

B.  The  cable  is  damaged.  Correct  answer. 

C.  The  battery  is  not  secure. 

D.  The  key  is  invalid. 

Question  Fades. 

Main  Player  Begins  Radio  Interaction  2  -  In  center  view  of  the  screen,  the  radio 
connected  to  the  SKL  device  is  shown.  It’s  shown  that  there  is  an  error  message  on  the  LCD 
screen.  The  main  player  continues  to  have  control  of  radio  using  mouse  and  keypad.  A  clock  is 
displayed  at  the  bottom  right  comer  of  the  screen,  continuing  the  countdown  of  15  minutes  from 
previous  interaction.  Players  can  hear  each  other  via  headsets. 


Option  1  for  Radio  Interaction:  The  player  is  able  to  disconnect  the  keyfill  cable  and  replace  it 
with  a  spare  cable  ( from  his  toolkit).  Then  he  will  access  Keyfill  Screen  by  pressing 
ALT+MODE  and  press  PTT  to  load  keys.  This  is  the  correct  answer.  Show  radio  LCD  saying 
“Lill  Success.” 
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Option  2  for  Radio  Interaction:  The  player  does  not  replace  the  keyfill  cable.  This  is  an 
incorrect  answer. 


NOTE:  For  the  inactive  players,  all  above  options  have  a  top  view  (as  if  looking  over  one’s 
shoulder). 


Incorrect  Answer  /  Time  Runs  out:  Display  text  “Correct  action:  Replace  the  damaged 
keyfill  cable.”  Text  fades. 
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TRAIN  II  Assessments 
Collaborative  Scenario  -  Vignette  Script 

VIGNETTE  6  -  MAIN  PLAYER  VIEW 

Starting  position  -  Scene  begins  by  showing  a  small  village  in  the  desert.  The  three  players  and 
Colonel  are  walking  towards  a  school  house.  Player  2  (RTO)  and  Player  3  are  equipped  with  a 
radio  that  is  attached  with  an  external  MIC  (clipped  to  shoulder).  Player  1  is  equipped  with  a 
radio  without  the  external  MIC. 


Start  Action  1  -  Start  scene  with  side  scrolling  view.  All  three  players  and  Colonel  are 
walking  towards  a  school  house.  Key  player  looks  down  at  his  radio  upon  hearing  a  sound. 


Sounds  -  Play  audible  indicator  sound  that  denotes  low  battery. 


Background  image:  At  the  bottom  of  the  screen,  Image  of  three  players,  the  main  player  (Player 
1)  is  highlighted  with  “You”  just  below  the  soldier  image. 


Enter  Text  -  Your  convoy  arrives  at  the  Village.  The  Colonel  is  scheduled  to  meet  with  an 
Official  to  review  the  School  House  project.  As  you  walk  toward  the  school,  you  hear  a  sound 
from  your  radio. 


Enter  Knowledge  Check  -  What  does  the  audible  indicator  from  your  radio  mean? 


A.  Transmit  timeout  warning. 

B.  Low  battery.  Correct  answer 

C.  Crypto  alarm. 

D.  Error  alarm. 

Question  Fades. 

Enter  Text.  You  now  have  3  minutes  to  address  the  audible  indicator.  Text  fades. 
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Begin  Radio  Interaction  1  -  In  center  view  of  the  screen,  the  radio  appears  fully 
assembled.  The  main  player  now  has  control  of  radio  using  mouse  and  mainpad.  A  clock  is 
displayed  at  the  bottom  right  comer  of  the  screen,  ticking  down  3  minutes.  Players  can  hear  each 
other  via  headsets. 
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Option  1  for  Radio  Interaction:  The  player  is  able  to  utilize  the  mouse  and  virtual  mainpad  to 
change  the  battery.  The  player  must  power  off  the  radio,  detach  the  old  battery,  and  attach  the 
spare  charged  battery  (To  attach  the  External  Battery  to  the  JEM,  hold  the  Battery 
perpendicular  to  the  bottom  of  the  radio.  Twist  the  Battery  clockwise  90  degrees  or  until  it  is 
locked  into  position ),  continue  start-up  procedures  and  then  attempt  to  establish  communications 
with  truck  ( correct  answer  stops  at  push  to  talk). 

The  player  must  select  this  option.  This  is  the  correct  answer. 


Option  2  for  Radio  Interaction:  Player  tries  to  remove  the  battery  without  first  powering  it  off; 
player  attempts  to  reprogram  the  radio;  the  player  attempts  to  remove  the  antenna;  the  player 


attempts  to  program  the  radio  for  keyfill,  etc.  These  are  incorrect  answers. 


Incorrect  Answer  /  Time  Runs  out  /  Partial  Credit:  Display  animated  video  of 
changing  the  battery  and  establishing  communication. 
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VIGNETTE  6  -  INACTIVE  PLAYER  VIEW 


Starting  position  -  Scene  begins  by  showing  a  small  village  in  the  desert.  The  three  players  and 
Colonel  are  walking  towards  a  school  house.  Player  2  (RTO)  and  Player  3  are  equipped  with  a 
radio  that  is  attached  with  an  external  MIC  (clipped  to  shoulder).  Player  1  is  equipped  with  a 
radio  without  the  external  MIC. 


Start  Action  1  -  Start  scene  with  side  scrolling  view.  All  three  players  and  Colonel  are 
walking  towards  a  school  house.  Key  player  looks  down  at  his  radio  upon  hearing  a  sound. 


Sounds  -  Play  audible  indicator  sound  that  denotes  low  battery. 


Background  image:  At  the  bottom  of  the  screen,  Image  of  three  players,  the  inactive  player 
(Player  2  /  RTO  or  Player  3)  is  highlighted  with  “You”  just  below  the  soldier  image. 


Enter  Text  -  Your  convoy  arrives  at  the  Village.  The  Colonel  is  scheduled  to  meet  with  an 
Official  to  review  the  School  House  project.  As  you  walk  toward  the  school,  you  hear  a  sound 
from  your  teammate’s  radio. 


Enter  Knowledge  Check  -  What  does  the  audible  indicator  from  your  teammate’s  radio  mean? 


A.  Transmit  timeout  warning. 

B.  Low  battery.  Correct  answer 

C.  Crypto  alarm. 

D.  Error  alarm. 

Question  Fades. 

Main  Player  Begins  Radio  Interaction  1  -  In  center  view  of  the  screen,  the  radio 
appears  fully  assembled.  The  main  player  now  has  control  of  radio  using  mouse  and  mainpad. 

A  clock  is  displayed  at  the  bottom  right  corner  of  the  screen,  ticking  down  60  seconds.  Players 
can  hear  each  other  via  headsets. 

Option  1  for  Radio  Interaction:  The  main  player  is  able  to  utilize  the  mouse  and  virtual 
mainpad  to  change  the  battery.  The  player  must  power  off  the  radio,  detach  the  old  battery,  and 
attach  the  spare  charged  battery  (To  attach  the  External  Battery  to  the  JEM,  hold  the  Battery 
perpendicular  to  the  bottom  of  the  radio.  Twist  the  Battery  clockwise  90  degrees  or  until  it  is 
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locked  into  position),  continue  start-up  procedures  and  then  attempt  to  establish  communications 
with  truck  ( correct  answer  stops  at  push  to  talk). 

The  player  must  select  this  option.  This  is  the  correct  answer. 

Option  2  for  Radio  Interaction:  Player  tries  to  remove  the  battery  without  first  powering  it  off; 
player  attempts  to  reprogram  the  radio;  the  player  attempts  to  remove  the  antenna,  etc.  These 
are  incorrect  answers. 

NOTE:  For  the  inactive  players,  all  above  options  have  a  top  view  (as  if  looking  over  one’s 
shoulder). 


Incorrect  Answer  /  Time  Runs  out  /  Partial  Credit:  Display  animated  video  of 
changing  the  battery  and  establishing  communication. 
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TRAIN  II  Assessments 
Collaborative  Scenario  -  Vignette  Script 

VIGNETTE  7  -  MAIN  PLAYER  VIEW 

Starting  position  -  (Picking  up  from  Vignette  6  —  dismounted  from  the  vehicle  and  at  the 
village).  Top  view  of  three  players  dismounted  outside  of  the  School  House.  The  three  players 
are  waiting  for  the  Colonel,  who  is  inside  of  the  School  House  in  a  meeting.  Player  2  (RTO)  and 
Player  3  are  equipped  with  a  radio  that  is  attached  with  an  external  MIC  (clipped  to  shoulder). 
Player  1  is  equipped  with  a  radio  without  the  external  MIC. 


Start  Action  1  -  Start  scene  with  side  scrolling  view.  The  Colonel  exits  the  School  House 
and  approaches  the  Truck  Commander. 


Sounds  -Play  Colonel  saying,  “The  inspection  is  complete.  Radio  back  to  the  Truck  and 
let  them  know  we’re  returning.” 


Continue  Action  1  -  The  main  player  (first-person  view)  grabs  his  radio  and  pushes  PTT. 
While  trying  to  make  communication,  he  does  not  hear  anything  on  the  radio.  He  turns  the 
channel  and  attempts  to  establish  communication  with  another  player  nearby  but  is  unsuccessful. 


Sounds  -  Play  other  player  saying,  “I  don’t  hear  anything  from  you.” 


Background  image:  Black  and  white  faded  image  of  the  main  player  looking  at  the  radio.  At 
the  bottom  of  the  screen,  image  of  three  players,  the  main  player  (Player  3)  is  highlighted  with 
“You”  just  below  the  soldier  image. 


Enter  Text  -  When  trying  to  make  communications  with  the  truck,  you  were  unable  to  establish 
communication.  Text  fades. 

Enter  Knowledge  Check  -  Select  the  most  likely  reason(s)  why  you  were  unable  to  reach  the 
truck  on  your  radio? 

M.  The  battery  screws  are  loose. 

N.  The  o-ring  on  the  battery  is  damaged. 

O.  The  antenna  is  damaged.  Correct  answer. 

P.  The  antenna  is  not  properly  connected.  Correct  answer. 

Question  Fades. 

Enter  Text  -  You  now  have  5  minutes  to  solve  this  problem.  Text  fades. 
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Begin  Radio  Interaction  1  -  In  center  view  of  the  screen,  the  radio  appears  fully 
assembled.  The  main  player  now  has  control  of  radio  using  mouse  and  mainpad.  A  clock  is 
displayed  at  the  bottom  right  comer  of  the  screen,  ticking  down  5  minutes.  Players  can  hear  each 
other  via  headsets. 


Option  1  for  Radio  Interaction:  The  player  is  able  to  disconnect  the  antenna,  examines  it, 
discovers  that  there’s  dirt  in  the  connector,  cleans  it  out  and  reattaches  the  antenna. 

The  player  must  select  this  option.  This  is  the  correct  answer. 

Option  2  for  Radio  Interaction:  Player  tries  to  remove  the  battery;  player  attempts  to 
reprogram  the  radio;  the  player  attempts  to  power  off  the  radio,  etc.  These  are  incorrect 
answers. 

« 

Incorrect  Answer  /  Time  Runs  out:  Display  text:  “Correct  action:  Clean  out  the  antenna 
connector.”  Text  fades.  May  include  any  available  clips  of  dirty  vs.  clean  antenna 
connectors. 


VIGNETTE  7  -  INACTIVE  PLAYER  VIEW 

Starting  position  -  (Picking  up  from  Vignette  6 — dismounted  from  the  vehicle  and  at  the 
village).  Top  view  of  three  players  dismounted  outside  of  the  School  House.  The  three  players 
are  waiting  for  the  Colonel,  who  is  inside  of  the  School  House  in  a  meeting.  Player  2  (RTO)  and 
Player  3  are  equipped  with  a  radio  that  is  attached  with  an  external  MIC  (clipped  to  shoulder). 
Player  1  is  equipped  with  a  radio  without  the  external  MIC. 


Start  Action  1  -  Start  scene  with  side  scrolling  view.  The  Colonel  exits  the  School  House 
and  approaches  the  Truck  Commander. 


0 


Sounds  -  Play  Colonel  saying,  “The  inspection  is  complete.  Radio  back  to  the  Truck  and 
let  them  know  we’re  returning.” 


Continue  Action  1  -  The  main  player  (second-person  view)  grabs  his  radio  and  pushes 
PTT.  While  trying  to  make  communication,  he  does  not  hear  anything  on  the  radio.  He  turns  the 
channel  and  attempts  to  establish  communication  with  another  player  nearby  but  is  unsuccessful. 
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Sounds  -  Play  other  player  saying,  “I  don’t  hear  anything  from  you.” 

Background  image:  Black  and  white  faded  image  of  the  main  player  looking  at  the  radio.  At 
the  bottom  of  the  screen,  image  of  three  players,  the  inactive  player  (Player  1  or  Player  2  /  RTO) 
is  highlighted  with  “You”  just  below  the  soldier  image. 


Enter  Text  -  When  trying  to  make  communications  with  the  truck,  your  teammate  was  unable  to 
establish  communication.  Text  fades. 

Enter  Knowledge  Check  -  Select  the  most  likely  reason(s)  why  your  teammate  was  unable  to 
reach  the  truck  on  your  radio? 

A.  The  battery  screws  are  loose. 

B.  The  o-ring  on  the  battery  is  damaged. 

C.  The  antenna  is  damaged.  This  is  the  correct  answer. 

D.  The  antenna  is  not  properly  connected.  This  is  the  correct  answer. 

Question  Fades. 

Enter  Text  -  The  main  player  now  have  5  minutes  to  solve  this  problem.  Text  fades. 


Main  Player  Begins  Radio  Interaction  1  -  In  center  view  of  the  screen,  the  radio 
appears  fully  assembled.  The  main  player  now  has  control  of  radio  using  mouse  and  mainpad. 
A  clock  is  displayed  at  the  bottom  right  corner  of  the  screen,  ticking  down  5  minutes.  Players 
can  hear  each  other  via  headsets. 


Option  1  for  Radio  Interaction:  The  player  is  able  to  disconnect  the  antenna,  examines  it, 
discovers  that  there’s  dirt  in  the  connector,  cleans  it  out  and  reattaches  the  antenna. 

The  player  must  select  this  option.  This  is  the  correct  answer. 


Option  2  for  Radio  Interaction:  Player  tries  to  remove  the  battery;  player  attempts  to 
reprogram  the  radio;  the  player  attempts  to  power  off  the  radio,  etc.  These  are  incorrect 
answers. 


NOTE:  For  the  inactive  players,  all  above  options  have  a  top  view  (as  if  looking  over  one’s 
shoulder). 


Incorrect  Answer  /  Time  Runs  out:  Display  text:  “Correct  action:  Clean  out  the  antenna 
connector.”  Text  fades.  May  include  any  available  clips  of  dirty  vs.  clean  antenna 
connectors. 
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JEM  Training  Research  Assessment  Project 


Beta  Test  Directions  &  Feedback  Form 
January  2013 


Developed  for: 

Army  Research  Institute 
Orlando,  FL 

In  conjunction  with  the  Fort  Gordon  Signal  School 
Augusta,  GA 


Prepared  By: 

ICF  International 
9300  Lee  Highway 
Fairfax,  VA  22031 
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What  is  the  purpose  of  the  JEM  Training  Research  Assessment  Project? 

Over  the  past  year,  the  Army  Research  Institute  (ARI)  and  contracting  firm  ICF  International 
(ICF)  have  developed  prototypes  for  training  Soldiers  to  use  the  JTRS  Enhanced  MBITR  (JEM). 
The  prototypes  include  trainings  on  a  mobile  device  (tablet),  a  virtual  classroom  (on  computer), 
and  a  collaborative,  scenario-based  exercise  (also  on  a  computer).  Throughout  the  trainings, 
Soldiers  are  required  to  complete  a  number  of  different  assessments  to  assess  knowledge  and 
learning.  The  primary  goal  of  this  project  is  to  conduct  research  on  training  environments  and 
assessments  that  are  more  Soldier-centered  than  most  traditional  Army  training. 

What  is  the  purpose  of  the  beta  test? 

For  this  beta  test,  ARI  and  ICF  would  like  to  have  Soldiers  try  out  the  training  and  assessment 
prototypes.  Specifically,  you  will  be  asked  to  complete  all  three  trainings:  the  JEM  Training 
Mobile  Application,  JEM  Training  Virtual  Classroom,  and  JEM  Training  Collaborative 
Assessment.  Within  each  of  these  trainings  you  will  also  be  asked  to  complete  a  number  of 
assessments.  Your  performance  on  the  training  and  assessments  will  not  be  connected  to  your 
name  in  any  way;  the  purpose  of  the  beta  test  is  to  gather  your  feedback  on  the  content  of  the 
training  and  assessment,  as  well  as  the  usability  of  the  technology.  Data  collected  will  be  used 
for  research  purposes  and  to  improve  the  content  and  user  experience  during  the  trainings. 

In  this  packet,  we  have  provided  information  about  each  JEM  training  and  Feedback  Forms  for 
you  to  record  your  comments  as  you  proceed  through  the  trainings.  ARI  and  ICF  staff  will  be 
guiding  you  through  the  trainings  and  Feedback  Forms,  and  observing  throughout  the  day.  Upon 
completion  of  the  beta  test,  you  will  be  debriefed  and  asked  for  final  reactions  to  the  trainings. 

Who  is  the  target  audience  for  the  JEM  Training? 

The  target  audience  for  these  trainings  is  25Us  who  have  not  yet  taken  the  Advanced  Feadership 
Course. 

What  is  my  role  in  reviewing  the  trainings? 

We  are  sharing  these  prototypes  with  you  in  order  to  gather  feedback  on  the  general  appearance  / 
features,  functionality,  and  organization  of  the  trainings  and  assessments.  Refer  to  the  Feedback 
Forms  in  each  section  for  systematically  recording  your  comments. 

Thank  you  for  your  time  and  support. 
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Demographic  Questions 


TIME  IN  MILITARY 

How  many  total  years  of  military  service  have  you  completed?  (Include  time  in  current  and 
previous  tours  and  services) 

_ years 


RANK 

What  is  your  current  rank  or  grade? 


( )  CW5 

( )  CSM 

( )  SSG 

( )  CW4 

( )  SGM 

( )  SGT 

( )  CW3 

( )  1SG 

()SPC 

( )  CW2 

( )  MSG 

( )  PFC 

()  WOl 

( )  SFC 

( )  PV2 
()PV1 

TIME  IN  RANK 

How  many  months  have  you  served  in  your  current  rank,  grade,  or  pay  level? 
_ months 


MOS 

What  is  your  Military  Occupational  Specialty  (MOS)? 

( )  25U  Signal  Support  Systems  Specialist 

( )  Other  (please  specify): _ 

DEPLOYMENTS 

How  many  times  have  you  been  deployed?  Include  current  deployment  if  applicable.  Enter  a 
whole  number  (e.g.,  2). 

Number  of  deployments: _ 
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RADIO  TRAINING 

For  which  of  the  following  radios  have  you  received  training?  (check  all  that  apply) 


( )  AN/PRC-25 
( )  AN/PRC-77 

( )  AN/PRC-1 17F  (SATCOM) 

( )  S  INC  GARS 

( )  AN/PRC-148  V1/V2  (MBITR) 

( )  AN/PRC- 148  V3  or  newer  (JEM) 

( )  AN/PRC-152 
( )  AN/PRC- 154 

( )  Other(s)  -  Please  specify _ 

RADIO  EXPERIENCE  WHILE  DEPLOYED 

Which  of  the  following  radios  have  you  operated  while  deployed?  (check  all  that  apply) 


( )  AN/PRC-25 
( )  AN/PRC-77 

( )  AN/PRC-1 17F  (SATCOM) 

( )  S  INC  GARS 

( )  AN/PRC-148  V1/V2  (MBITR) 

( )  AN/PRC- 148  V3  or  newer  (JEM) 

( )  AN/PRC-152 
( )  AN/PRC- 154 

( )  Other(s)  -  Please  specify _ 

MBITR  EXPERIENCE 

Have  you  used  an  MBITR  or  JEM  prior  to  this  training?  If  yes,  how  many  months  of  experience 
do  you  have  using  an  MBITR  or  JEM? 

( )  Yes  with _ months  of  experience 

( )  No 
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MBITR/JEM  EXPERTISE 

Rate  your  level  of  expertise  using  an  MBITR/JEM  (choose  one  answer  that  best  describes  your 
expertise). 

( )  I  have  never  used  an  MBITR/JEM 

( )  I  have  used  an  MBITR/JEM  for  only  standard  communication 

( )  I  have  used  an  MBITR/JEM  and  performed  simple  troubleshooting  in  usual  circumstances 
to  ensure  I  was  able  to  communicate  with  others 

( )  I  have  used  an  MBITR/JEM  and  performed  advanced  troubleshooting  in  unusual 
circumstances  that  involved  adapting  and  making  complex  decisions 

EXPERIENCE  LOADING  KEYS 

Rate  your  level  of  experience  loading  keys  on  an  MBITR/JEM  (choose  one  answer  that  best 
describes  your  experience). 

( )  I  have  never  performed  a  keyfill  on  an  MBITR/JEM  and  have  never  seen  it  being 
performed 

( )  I  have  never  performed  a  keyfill  on  an  MBITR/JEM  but  I  have  seen  it  being  performed 
( )  I  have  performed  a  keyfill  on  an  MBITR/JEM  but  it  is  not  part  of  my  usual  duties 
( )  I  have  performed  a  keyfill  on  an  MBITR/JEM  and  it  is  part  of  my  usual  duties 

Information  about  the  Mobile  Training 

Structure 

The  JEM  mobile  training  covers  radio  assembly,  configuration,  programming,  operation,  and 
troubleshooting.  The  training  begins  with  a  pre-test  consisting  of  a  series  of  questions  designed 
to  assess  your  prerequisite  knowledge.  Following  the  pre-test,  you  will  access  the  training.  It 
includes  a  site  map  that  guides  you  through  the  training  material.  When  you  select  a  topic,  expect 
the  following  to  occur: 

1.  You  will  be  introduced  to  training  content  in  both  written  and  auditory  format. 

2.  You  may  be  shown  images  or  animated  sequences  on  the  screen  (depending  on  the  topic). 

3.  You  will  be  able  to  select  additional  content  or  examples  for  the  topic  by  clicking  those 
named  buttons.  This  content  is  only  in  written  format. 

4.  For  certain  topics,  you  will  have  the  opportunity  to  practice  using  a  simulated  radio. 

These  practice  opportunities  are  not  scored. 

5.  If  you  have  trouble  with  the  practice,  you  can  view  a  video  demonstration 

Assessments 

You  will  participate  in  the  following  types  of  assessments  in  the  Mobile  Application: 

1.  Computer  Adaptive  Pre-test  prior  to  this  training 
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2.  Interim  Check  on  Learning  Assessments  throughout  this  training 

3.  Computer  Adaptive  Post-test  upon  completion  of  this  training 

Navigate  through  the  training 

The  training  includes  a  navigation  screen  that  will  assist  users  in  navigating  the  training 
as  seen  below: 


Document  any  issues  or  comments  you  may  have  on  the  attached  JEM  Mobile  Training 
Feedback  Form 

Utilize  the  JEM  Mobile  Training  Feedback  Form  attached  to  provide  comments  on  your 
experience  with  the  Mobile  training.  Document  any  issues  that  you  encountered. 
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JEM  Mobile  Training  Feedback  Form 


Instructions  for  Completion 

Please  read  through  these  questions  and  keep  them  in  mind  while  participating  in  the  Mobile  training.  During 
the  training,  note  any  problems  or  errors  related  to  these  questions  and  describe  them  in  the  blank  pages  that 
follow. 

Technical  Issues 

Did  you  experience  any  problems  accessing  the  training? 

Did  you  experience  any  problems  during  the  training  with  accessing  content? 

Did  you  experience  any  problems  during  the  training  with  interaction  with  the  simulated  radio? 

Did  you  experience  any  problems  when  completing  assessment  items  during  the  training? 

Content  and  User  Practice 

Was  the  instructional  material  in  this  lesson  accurate  and  current? 

Was  the  simulated  radio  depicted  in  an  accurate  and  current  manner? 

Was  the  functionality  of  the  simulated  radio  accurate? 

Was  the  content  of  the  assessment  items  clear? 

Instructional  Design 

Was  the  material  presented  in  a  logical  sequence  to  help  you  understand  the  subject  matter? 

Were  the  assessment  items  during  the  training  relevant  to  the  training  content  previously  presented? 

Usability 

After  the  pre-test,  were  you  able  to  get  started  with  the  training  with  minimum  assistance? 

Were  you  able  to  successfully  operate  the  simulated  radio  during  practice  opportunities? 

Assessment  items  were  presented  in  various  formats.  Did  you  have  trouble  understanding  or  responding  to  any 
of  the  formats? 

Media 

Did  the  media  (i.e.,  graphics  and  animated  sequences)  appropriately  illustrate  the  points  being  presented? 

Did  the  simulated  radio  appropriately  illustrate  aspects  of  the  radio  assembly,  configuration,  programming,  and 
operation? 

Was  the  media  (i.e.,  graphics,  audio  clips,  animated  sequences)  included  in  assessment  items  appropriate? 
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JEM  Mobile  Training  Feedback  Form 


Rank/Title/Position:  _  Date  of  Review: 

JEM  Training  Mobile  Application 


Lesson  Name 

Detailed  Description  of  Feature  /  Issue 

C-9 


Lesson  Name 


Detailed  Description  of  Feature  /  Issue 


Information  about  t 

le  Virtual  Classroom 

Structure 

The  JEM  Virtual  Classroom  training  covers  radio  assembly,  configuration,  programming, 
operation,  and  troubleshooting.  You  will  find  some  overlap  in  content  from  the  Mobile  training; 
however,  the  Virtual  Classroom  allows  you  to  interact  with  an  instructor  and  to  have  more 
elaborate  interactions  with  the  simulated  radio  than  you  had  in  the  Mobile  training.  During  the 
Virtual  Classroom,  expect  the  following  to  occur: 

1.  Your  Virtual  Classroom  instructor  will  be  sharing  his  /  her  screen,  and  it  will  display  on 
your  screen. 

2.  You  may  be  shown  images  or  animated  sequences  on  the  screen  (depending  on  the  topic). 

3.  You  will  be  able  to  indicate  that  you  have  a  question  for  the  Virtual  Classroom  Instructor. 

4.  If  you  have  a  question  or  have  trouble  understanding  something  during  the  training,  you 
can  chat  with  your  Virtual  Instructor  or  seek  help  from  classroom  peers  using  the  chat 
function. 

5.  At  any  time,  you  can  access  a  Knowledge  Database  using  a  key  word  that  will  provide 
you  with  text  about  that  topic. 

Assessments 

You  will  participate  in  the  following  types  of  assessments  in  the  Virtual  Classroom: 

1.  Interim  Check  on  Learning  Assessments  throughout  this  training-  these  may  involve 
interaction  with  the  simulated  radio 

2.  Computer  Adaptive  Post-test  upon  completion  of  this  training 
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Navigate  through  the  training 

The  image  seen  below  is  the  user  interface  for  the  Virtual  Classroom: 


Document  any  issues  or  comments  you  may  have  on  the  attached  JEM  Virtual  Classroom 

Feedback  Form 

Utilize  the  JEM  Virtual  Classroom  Feedback  From  attached  to  provide  comments  on 
your  experience  with  the  Virtual  Classroom.  Document  any  issues  that  you 
encountered. 
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JEM  Virtual  Classroom  Feedback  Form 


Instructions  for  Completion 

Please  read  through  these  questions  and  keep  them  in  mind  while  participating  in  the  Virtual  Classroom.  During 
the  Virtual  Classroom,  note  any  problems  or  errors  and  describe  them  in  the  blank  pages  that  follow. 

Technical  Issues 

Did  you  experience  any  problems  entering  the  Virtual  Classroom? 

Did  you  experience  any  problems  during  the  training? 

Did  you  experience  any  problems  when  completing  assessment  items  during  the  training? 

Content  and  User  Practice 

Was  the  instructional  material  in  this  training  accurate  and  current? 

Was  the  simulated  radio  depicted  in  an  accurate  and  current  manner? 

Was  the  functionality  of  the  simulated  radio  accurate? 

Was  the  content  of  the  assessment  items  clear? 

Instructional  Design 

Was  the  material  presented  in  a  logical  sequence  to  help  you  understand  the  subject  matter? 

Were  the  assessment  items  during  the  training  relevant  to  the  training  content  previously  presented? 

Usability 

Were  you  able  to  use  the  Virtual  Classroom  functionalities  (i.e.  question  and  chat  function)  with  minimum 
assistance? 

Were  you  able  to  successfully  operate  the  simulated  radio  during  practice  opportunities? 

Assessment  items  were  presented  in  various  formats.  Did  you  have  trouble  understanding  or  responding  to  any 
of  the  formats? 

Media 

Did  the  media  (i.e.,  graphics  and  animated  sequences)  appropriately  illustrate  the  points  being  presented? 

Did  the  simulated  radio  appropriately  illustrate  aspects  of  the  radio  assembly,  configuration,  programming,  and 
operation? 

Was  the  media  (i.e.,  graphics,  audio  clips,  animated  sequences)  included  in  assessment  items  appropriate? 
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JEM  Virtual  Classroom  Feedback  Form 


Rank/Title/Position:  _  Date  of  Review: 

JEM  Training  Virtual  Classroom 


Lesson  Name 

Detailed  Description  of  Feature/Issue 
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Lesson  Name 

Detailed  Description  of  Feature/Issue 

C-14 


Information  about  the  Collaborative  Assessment 


Structure 

The  Collaborative  Assessment  places  you  and  two  other  team  members  in  real-to-life  situations 
involving  the  JEM  through  Virtual  Reality  experience.  The  assessment  is  based  on  a  single 
mission  with  7  different  situations  within  that  mission.  For  each  situation,  one  key  player  on  the 
team  will  be  actively  operating  the  simulated  radio,  while  the  other  two  team  members  will 
remain  available  to  assist  through  voice  chatting  capabilities.  During  the  Collaborative 
Assessment,  expect  the  following  to  occur: 

1.  You  and  your  team  will  be  viewing  the  same  screen  showing  the  Virtual  Reality  and 
active  player  avatar. 

2.  All  team  members  will  be  provided  a  situation  and  asked  to  answer  multiple  choice 
questions  individually  prior  to  the  active  player  engages  in  simulated  radio  manipulation 
tasks.  You  will  not  receive  feedback  on  this  question. 

3.  The  active  player  will  be  able  to  access  and  manipulate  the  simulated  radio  (through  a 
series  of  interfaces)  in  order  to  successfully  accomplish  the  task  at-hand.  Inactive  team 
members  will  not  be  able  to  access  and  manipulate  the  simulated  radio. 

4.  Each  task  will  have  a  specified  time  limit. 

5.  The  active  player  can  communicate  with  or  receive  verbal  assistance  from  other  team 
members. 

6.  The  active  player  must  click  the  “submit”  button  to  submit  his  or  her  action  or  he  or  she 
can  reset  any  actions  and  re-do  them  differently  if  the  submit  button  has  not  yet  been 
selected  and  time  has  not  yet  run  out. 

7.  If  the  submitted  action  is  correct,  your  team  moves  on  to  the  next  situation. 

8.  If  the  submitted  action  is  not  correct  or  you  run  out  of  time  before  submitting  the  action, 
your  team  will  see  an  animated  video  of  the  correct  answer  before  moving  on  to  the  next 
situation 
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Navigate  through  the  training 

The  image  seen  below  is  the  user  interface  for  the  collaborative  assessment: 


Document  any  issues  or  comments  you  may  have  on  the  attached  JEM  Collaborative 

Assessment  Feedback  Form 

Utilize  the  JEM  Collaborative  Assessment  Feedback  Form  attached  to  provide 
comments  on  your  experience  with  the  Collaborative  Scenario.  Document  any  issues 
that  you  encountered. 
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JEM  Collaborative  Assessment  Feedback  Form 


Instructions  for  Completion 

Please  read  through  these  questions  and  keep  them  in  mind  while  participating  in  the  Collaborative 
Assessment.  During  the  Collaborative  Assessment,  note  any  problems  or  errors  and  describe  them  in  the 
blank  pages  that  follow. 

Technical  Issues 

Did  you  experience  any  problems  accessing  the  Collaborative  Assessment? 

Did  you  experience  any  problems  during  the  assessment? 

Content  and  User  Practice 

Were  the  situations  presented  in  the  Collaborative  Assessment  appropriate  in  terms  of  accuracy  and 
feasibility? 

Was  the  simulated  radio  depicted  in  an  accurate  and  current  manner? 

While  manipulating  the  radio,  was  the  functionality  of  the  radio  accurate? 

Instructional  Design 

Were  the  situations  presented  in  a  logical  sequence  to  render  the  scenario  realistic? 

Were  you  prepared  to  perform  the  actions  presented  in  the  situations  based  on  what  you  had  learned  in 
the  Mobile  or  Virtual  Classroom? 

Usability 

Were  you  able  to  communicate  with  team  members  without  technical  issues? 

Were  you  able  to  successfully  operate  the  simulated  radio  when  you  were  the  active  player? 

Did  you  understand  what  you  needed  to  do  from  a  technical  standpoint  in  order  to  complete  each 
assessment?  For  example,  were  you  ever  in  a  situation  where  you  felt  you  knew  the  correct  answer  or 
solution  and  were  unable  to  execute  it  due  to  technical  issues? 

Did  you  find  access  to  the  various  interfaces  intuitive?  (radio  accessories  bag,  keypad,  etc.) 
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JEM  Collaborative  Assessment  Feedback  Form 


Rank/Title/Position:  _  Date  of  Review: 

JEM  Training  Collaborative  Assessment 


Scenario 

Detailed  Description  of  Feature/Issue 
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Scenario 


Detailed  Description  of  Feature/Issue 


Debriefing  Questions 


1.  Did  you  find  the  trainings  more  or  less  helpful  than  traditional  classroom  learning? 


2.  Did  you  find  the  technology  to  be  easy  to  understand  and  use? 


3.  What,  if  anything,  would  you  change  about  the  trainings  and  assessments? 


4.  Please  provide  any  additional  comments  you  have  about  the  trainings. 
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APPENDIX  D 

ONLINE  REACTION  QUESTIONNAIRE  ITEMS 
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Dimension 


Item 


Format 


Usability 

The  technology  interface  was  easy  to  use. 

5-point  disagree  /  agree  scale 

Usability 

The  technology  allowed  for  easy  review. 

5-point  disagree  /  agree  scale 

Usability 

I  was  able  to  access  the  training  with  minimum  assistance. 

5-point  disagree  /  agree  scale 

Usability 

I  was  able  to  successfully  operate  the  functionality  within  the  training. 

5-point  disagree  /  agree  scale 

Usability 

I  am  satisfied  with  the  technology  interface. 

55-point  disagree  /  agree  scale 

Content/ISD 

The  session  objectives  were  met. 

5-point  disagree  /  agree  scale 

Content/ISD 

The  material  in  this  lesson  was  accurate  and  current. 

5-point  disagree  /  agree  scale 

Content/ISD 

The  design  of  the  training  was  an  effective  way  to  present  the  subject  matter? 

5-point  disagree  /  agree  scale 

Content/ISD 

The  material  was  presented  in  a  logical  sequence  so  that  it  helped  me  understand 
the  subject  matter. 

5-point  disagree  /  agree  scale 

Content/ISD 

The  media  (i.e.,  graphics  and  animated  sequences)  appropriately  illustrate  the 
points  being  discussed. 

5-point  disagree  /  agree  scale 

Content/ISD 

The  design  and  presentation  of  material  motivated  me  to  learn. 

5-point  disagree  /  agree  scale 

Content/ISD 

Overall,  I  am  pleased  with  the  way  the  training  was  presented. 

5-point  disagree  /  agree  scale 

Perceived  Learning 

Learning  this  material  was  fun. 

5-point  disagree  /  agree  scale 

Perceived  Learning 

Please  rate  the  following: 

My  knowledge  of  the  subject  PRIOR  to  taking  this  lesson: 

5-point  poor  /  excellent  scale 

D-2 


My  knowledge  of  the  subject  AFTER  to  taking  this  lesson: 

5-point  poor  /  excellent  scale 

My  ability  to  apply  the  strategies  and  techniques  presented  to  an  actual 
situation  in  this  subject  area  PRIOR  to  taking  this  lesson: 

5-point  poor  /  excellent  scale 

My  ability  to  apply  the  strategies  and  techniques  presented  to  an  actual 
situation  in  this  subject  area  AFTER  to  taking  this  lesson: 

5-point  poor  /  excellent  scale 

Perceived  Learning 

To  what  do  you  attribute  the  differences  in  your  PRIOR  and  AFTER  responses? 

Open-ended 

Perceived  Learning 

Overall,  I  have  learned  a  lot  from  this  training. 

5-point  disagree  /  agree  scale 

Value/Utility 

It  is  clear  to  me  that  the  people  conducting  the  training  understand  how  I  will  use 
what  I  learn. 

5-point  disagree 

/  agree  scale 

Value/Utility 

This  training  was  relevant  to  my  job  in  the  Army. 

5-point  disagree 

/  agree  scale 

Value/Utility 

I  believe  the  training  will  help  me  do  my  current  job  in  the  Army  better. 

5-point  disagree 

/  agree  scale 

Value/Utility 

I  learned  something  I  can  apply  immediately  to  my  work  in  the  Army. 

5-point  disagree 

/  agree  scale 

Value/Utility 

I  plan  to  use  what  I  learned  on  my  job  in  the  Army. 

5-point  disagree 

/  agree  scale 

Value/Utility 

I  get  excited  when  I  think  about  trying  to  use  my  new  learning  on  my  job  in  the 
Army. 

5-point  disagree 

/  agree  scale 

Value/Utility 

The  training  materials  will  be  available  for  me  to  use  on  my  job  in  the  Army. 

5-point  disagree 

/  agree  scale 

Value/Utility 

I  will  be  using  the  equipment  on  my  job  in  the  Army  after  the  training. 

5-point  disagree 

/  agree  scale 

Value/Utility 

The  training  was  of  practical  value  to  me. 

5-point  disagree 

/  agree  scale 

Satisfaction 

I  enjoyed  this  training  program. 

5-point  disagree  /  agree  scale 

Satisfaction 

My  time  on  the  training  was  well  spent. 

5-point  disagree  /  agree  scale 
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Satisfaction 

I  would  recommend  this  training  program  to  other  Soldiers. 

5-point  disagree  /  agree  scale 

What,  if  anything,  would  you  change  about  the  training? 

Open-ended 

Please  provide  any  additional  comments  you  have  about  the  training. 

Open-ended 

Technical  Issues 

Did  you  experience  any  problems  accessing  the  training? 

Yes  /  No 

Technical  Issues 

If  yes,  please  describe. 

Open-ended 
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